Multi-sensor data fusion for automotive systems

ABSTRACT

A sensor fusion system associated with a vehicle includes a sensor interface communicatively coupled to a plurality of sensors in the vehicle and a vehicle experience system. The sensor interface comprises an input receiving data from each of the plurality of sensors and an output configured to output fused vehicle data based on the data received from the plurality of sensors. The vehicle experience system is coupled to the output of the sensors interface to receive the fused vehicle data. The vehicle experience system includes one or more processors and a non-transitory computer readable storage medium storing instructions that when executed by one or more processors cause the one or more processors to control at least one parameter of the vehicle based on the fused vehicle data.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Patent No.62/798,721, filed Jan. 30, 2019; U.S. Provisional Patent No. 62/937,414,filed Nov. 19, 2019; U.S. Provisional Patent No. 62/948,745, filed Dec.16, 2019; and U.S. Provisional Patent No. 62/957,727, filed Jan. 6,2020, each of which is incorporated herein by reference in its entirety.

This application is related to U.S. patent application Ser. No. ______,filed Jan. 30, 2020 (Attorney Docket No. 132220-8004.US02), U.S. patentapplication Ser. No. ______, filed Jan. 30, 2020 (Attorney Docket No.132220-8004.US03), U.S. patent application Ser. No. ______, filed Jan.30, 2020 (Attorney Docket No. 132220-8004.US04), and U.S. patentapplication Ser. No. ______, filed Jan. 30, 2020 (Attorney Docket No.132220-8004.US05),each of which is incorporated herein by reference inits entirety.

TECHNICAL FIELD

This disclosure relates to modifying features of a vehicle based ondetected characteristics of the environment surrounding the vehicle.

BACKGROUND

Vehicles are generally utilized by individuals for transportation tovarious destinations. For example, a vehicle can include a car, truck,train, airplane, etc. While vehicles are generally utilized fortransportation, vehicles include components configured to performvarious functionalities.

For example, a vehicle can include components configured to modify theplayback of entertainment content (e.g., audio, video, gaming content).As another example, the vehicle includes components configured tocontrol an internal environment of the vehicle. These components aregenerally included in a vehicle to increase a user experience while inthe vehicle.

In many cases, a user may modify various components of the vehicle usingvarious interfaces (an interactive display, a set of dials, buttons,etc.), where modification of these components are generally performedresponsive to detecting a modification to an interface performed by theuser.

Further, each user in a vehicle generally has a set of unique desiredsettings of components in the vehicle. For instance, a first user maywant audio playing from speakers in the vehicle at a first volume, whilea second user may want no audio playing and a set of fans in the vehicleoperating at a high speed.

Moreover, the set of unique desired settings for a user may differ basedon other aspects. For example, such aspects that can differ the set ofunique desired settings for the user can include an emotional state ofthe user, the time of day, external weather conditions, etc.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example environment that includes one or more vehicles andone or more remote servers.

FIG. 2 is a schematic diagram illustrating an example vehicle.

FIG. 3 is a block diagram illustrating components of a vehicle,according to some embodiments.

FIG. 4 is an example abstraction illustrating relationships betweencomponents and data associated with the vehicle.

FIG. 5 is a block diagram illustrating an example configuration of thevehicle experience system with respect to other components of thevehicle.

FIG. 6A is a flowchart illustrating a process to determine the driver'semotional state.

FIG. 6B illustrates example data types detected and generated during theprocess shown in FIG. 6A.

FIG. 7 is a flowchart illustrating another process for detecting anemotional state of a person, according to some embodiments.

FIG. 8 is an example flow diagram of a process for generatinguser-specific output actions in a vehicle.

FIG. 9 is an example block diagram for generating a user profile.

FIG. 10 is a block diagram of an example vehicle with multiple regions.

FIG. 11 is an example block diagram for generating personalized outputactions.

FIG. 12 is a block diagram of an example method for performing one ormore output actions relating to a modification of at least one featureof a vehicle.

FIG. 13 is a block diagram illustrating functional modules executed bythe media system, according to some embodiments.

FIG. 14 is a flowchart illustrating an example process for selectingentertainment content for output to a passenger in a vehicle.

FIG. 15 is a block diagram of an example method for generating andimplementing user-specific comfort-related output actions in a vehicle.

FIG. 16 is a block diagram of an example method for generating andimplementing user-specific safety-related output actions in a vehicle.

FIG. 17 is a block diagram illustrating an example of a processingsystem in which at least some operations described herein can beimplemented.

DETAILED DESCRIPTION Overview

Automotive vehicles have a wide variety of sensor hardware available andare continuously adding new capabilities as technology improves andcosts reduce. However, the data produced by the sensors is typicallytrapped in silos for single-use purpose, resulting an enormous universeof untapped data available in automotive vehicles. A vehicle experiencesystem uses these sensor inputs to create a personalized,contextually-aware and real-time adaptive experience for drivers and/orpassengers of vehicles. Various embodiments of the vehicle experiencesystem described herein can measure an emotional state of a driver orpassenger in a vehicle, as well as context of the vehicle or itsenvironment, and control entertainment, safety, or comfort systems ofthe vehicle in manner that is dynamically responsive to the person'semotional state, the context of the vehicle or environment, or acombination thereof. To control the systems of the vehicle, variousembodiments of the vehicle experience system create, train, and utilizepersonalized models that learn a person's emotional states, habits, andreactions over time to improve the ability of the vehicle to respond toor anticipate a person's needs.

Some embodiments of the vehicle experience system are described hereinwith respect to vehicles, including measuring parameters in vehicles andcontrolling outputs associated with the vehicles to create dynamic,personalized vehicle experiences. However, the experience systemdescribed herein is not limited to vehicles. Similar methodologies andsystems can be implemented in other environments in which people spendtime or pass through, including homes, office buildings, hotels,restaurants, transit stations, or schools.

System Overview

The present embodiments relate to vehicle passenger experience andpersonalization. Embodiments of systems and processes described hereincreate a personalized, adaptable experience in a vehicle that can bebased on data fused from multiple sensors in a vehicle.

Environment Overview

FIG. 1 is an example environment 100 that includes one or more vehicles110 and one or more remote servers 120.

The vehicle 110 can include any vehicle capable of carrying one or morepassengers, including any type of land-based automotive vehicle (such ascars, trucks, or buses), train, flying vehicle (such as airplanes,helicopters, or space shuttles), or aquatic vehicle (such as cruiseships). The vehicle 110 can be a vehicle operated by any driving mode,including fully manual (human-operated) vehicles, self-driving vehicles,or hybrid-mode vehicles that can switch between manual and self-drivingmodes. As used herein, a “self-driving” mode is a mode in which thevehicle 110 operates at least one driving function in response toreal-time feedback of conditions external to the vehicle 110 andmeasured automatically by the vehicle 110. The driving functions caninclude any aspects related to control and operation of the vehicle,such as speed control, direction control, or lane positioning of thevehicle 110. To control the driving functions, the vehicle 110 canreceive real-time feedback from external sensors associated with thevehicle 110, such as sensors capturing image data of an environmentaround the vehicle 110, or sources outside the vehicle 110, such asanother vehicle or the remote server 120. The vehicle 110 can processthe sensor data to, for example, identify positions and/or speeds ofother vehicles proximate to the vehicle 110, track lane markers,identify non-vehicular entities on the road such as pedestrians or roadobstructions, or interpret street signs or lights. In some cases, thevehicle 110 operates in an autonomous mode under some drivingcircumstances, such that the driver does not need to control any drivingfunctions during the autonomous operation. In other cases, the vehicle110 controls one or more driving functions while the driver concurrentlycontrols one or more other driving functions.

The vehicle 110 can have a regular driver, or a person who is usuallydriving the vehicle when the vehicle is operated. This person may, forexample, be an owner of the vehicle 110. In other cases, the vehicle 110can be a shared vehicle that does not have a regular driver, such as arental vehicle or ride-share vehicle.

In some embodiments, a vehicle 110 can retrieve a user profile that isassociated with a user that primarily operates the vehicle. In otherembodiments, upon detecting a user in the vehicle (e.g., by anindication from a mobile device, facial recognition), a unique userprofile associated with the user can be retrieved. Based on the userprofile, user-specific output actions can be performed that modifyvarious settings in the vehicle.

The remote server 120 can process or store data received from thevehicle 110, or can send data to the vehicle 110. In variousembodiments, the remote server 120 can include data processing andanalysis functionality to, for example, determine a context of aninterior or exterior of the vehicle 110. Other embodiments of the remoteserver 120 can communicate content to the vehicle 110, including mediacontent or remote updates to software or configuration settings of thevehicle 110. The remote server 120 can be configured as a physicalserver, a cloud computing platform, or any other suitable configuration.

The remote server 120 and vehicle 110 can communicate over one or morenetwork channels 130, which can include any of a variety of individualconnections via the internet such as cellular or other wirelessnetworks, such as 4G networks, 5G networks, or WiFi. In someembodiments, the network may connect terminals, services, and mobiledevices using direct connections such as radio-frequency identification(RFID), near-field communication (NFC), Bluetooth™, low-energyBluetooth™ (BLE), WiFi™, ZigBee™, ambient backscatter communications(ABC) protocols, USB, or LAN. Because the information transmitted may bepersonal or confidential, security concerns may dictate one or more ofthese types of connections be encrypted or otherwise secured. In someembodiments, however, the information being transmitted may be lesspersonal, and therefore the network connections may be selected forconvenience over security. The network may comprise any type of computernetworking arrangement used to exchange data. For example, the networkmay be the Internet, a private data network, virtual private networkusing a public network, and/or other suitable connection(s) that enablescomponents in system environment 100 to send and receive informationbetween the components of system environment 100. The network may alsoinclude a public switched telephone network (“PSTN”) and/or a wirelessnetwork.

The vehicles 110A, 110B can also communicate with each other bycommunicating either indirectly through the remote server 120 ordirectly over a vehicle-to-vehicle communications channel 130.

Vehicle Environment Overview

FIG. 2 is a schematic diagram illustrating an example vehicle 110. Thevehicle 110 can include one or more seats 230 in which passengers can beseated while they ride in the vehicle. By way of example, FIG. 2illustrates a driver seat 230A and three passenger seats 230B, 230C, and230D, but the vehicle 110 can have any number of seats and can configurethe seats differently. Further, in FIG. 2, the vehicle 110 can include avehicle zone 235 that includes a region that includes the interior ofthe vehicle.

As shown in FIG. 2, the vehicle 110 can have one or more displays 205.The one or more displays 205 can include a dedicated, persistentdisplay, such as a display associated with an infotainment system of thevehicle 110 or a head-up display positioned within a field of vision ofa driver while the driver is driving the vehicle 110. Additionally oralternatively, the one or more displays 205 can include devices that areused to display content only under some circumstances. For example, anyglass in the vehicle—such as a windshield, one or more side windows, ora rearview or side mirror—can be activated to be used as a displaydevice 205. When the display features of the glass are not activated,the glass can be, for example, transparent, to allow passengers to seethrough the glass to an exterior of the vehicle, or reflective, toenable passengers to view behind or to the side of the vehicle.

In some embodiments, the display 205 is a head-up display positioned inor near the driver's field of view to display content in a manner thatis visible but does not obstruct the driver's view of the road whiledriving. For example, the head-up display may be positioned near abottom of a windshield. Alternatively, the head-up display may bepositioned near a top of the windshield or on a side of the windshieldaway from a central field of view while the driver is looking forward.Furthermore, the head-up display may be integrated into the windshieldor a display device separate from the windshield. The head-up displaymay be translucent or transparent to reduce obstruction. In someembodiments, the head-up display displays content related to driving ofthe vehicle 110 while the vehicle is being driven. For example, thehead-up display can display speed, navigation instructions, ameasurement of a distance to the next car, or fuel level or batterycharge status.

In some embodiments, the head-up display has a first size while thevehicle 110 is being driven, and a second size when the vehicle 110 isnot being driven. Alternatively, content can be displayed within an areaof the head-up display that has a first size while the vehicle 110 isbeing driven, while the content can be displayed in an area that has asecond size when the vehicle is not being driven. For example, if adriver is consuming media content while recharging a battery of thevehicle 110, the head-up display can be changed to a stationary mode inwhich a larger portion of the windshield is used to the display themedia content. The larger portion of the windshield can include anentirety of the windshield, or a portion of the windshield that islarger than the portion used by the head-up display while the vehicle isdriven. In some embodiments, a size of a display can change based on anoperation state of the vehicle. For example, when the vehicletransitions from an operating state to an idle state, the size of thedisplay can be increased.

The vehicle 110 can have an entertainment system 210. The entertainmentsystem 210 can provide entertainment content to passengers in thevehicle 110, and can include one or more output devices in the vehiclethat are configured to output the entertainment content to thepassengers. The output devices can include the displays 205 to displayvideo, image, or textual content and a sound system including one ormore speakers to play audio content. In some cases, the output devicesassociated with the entertainment system 210 can further includelighting systems in the interior of the vehicle or tactile outputdevices such as a vibrating device in the seats 230. The entertainmentsystem 210 can output entertainment content for an entire interior ofthe vehicle 110 at the same time, or can provide different content fordifferent zones in the vehicle 110. For example, referring to FIG. 2,the entertainment system 210 can output entertainment content for thevehicle zone 235, or can output first entertainment content for a personin the driver seat 230A and second entertainment content for a person inthe passenger seat 230B. Some embodiments of the entertainment system210 automatically recommend and/or select entertainment content foroutput to a user.

The vehicle 110 can include a plurality of sensors 215 configured togenerate data related to parameters inside the vehicle 110 and outsidethe vehicle 110, including parameters related to one or more passengersinside the vehicle. Example parameters that can be measured by thesensors 215 include vehicle speed, acceleration, lane position, steeringangle, in-cabin decibel level, audio volume level, current informationdisplayed by a multimedia interface in the vehicle, force applied by theuser to the multimedia interface, ambient light, humidity level, rawvideo feed (whether from sources internal or external to the vehicle),audio input, user metadata, user state, calendar data, userobservational data, contextual external data, traffic conditions,weather conditions, in-cabin occupancy information, road conditions,user drive style, or non-contact biofeedback. Thus, example sensors 215include internal or external cameras, eye tracking sensors, temperaturesensors, audio sensors, weight sensors in a seat 230, force sensorsmeasuring force applied to devices such as a steering wheel or display205, accelerometers, gyroscopes, light detecting and ranging (LIDAR)sensors, or infrared sensors. Some of the sensors 215 can, at least inpart, be used for biosensing, such as identifying that one or morepeople or animals are in a vehicle, determining where the people,objects, or animals are located in the vehicle, or identifying emotionalstates of a person in a vehicle.

The vehicle 110 can include one or more input devices 220. At least oneof the input devices 220 can be a device configured to receive a tactileinput from a passenger. For example, the vehicle 110 may include aninfotainment system with soft or hard buttons the user can touch toprovide an input into the vehicle. One or more of the sensors 215 canadditionally or alternatively be used as an input device 220. Forexample, one of the sensors 215 can be a camera configured to captureimage data of a passenger that can be input to a gaze tracking tool. Thepassenger's gaze, as tracked by the gaze tracking tool, can be used asan input signal to control aspects of the vehicle 110. For example, thevehicle 110 can determine that a passenger has selected an itemdisplayed on the display 205 by determining that the passenger islooking at the displayed item. Similarly, gesture tracking, voicerecognition, or biometric systems can be used as input devices 220 thatenable a passenger to provide explicit inputs to the vehicle 110 to makeselections or adjust parameters of the vehicle 110. The vehicle 110 canhave one or more input devices 220 associated with a passenger.

Some of the input devices 220 can include portions of the vehicle 110that can be used in a first state to control an aspect of the vehiclerelated to driving the vehicle, while also being usable in a secondstate to provide input to the media system. For example, the vehicle 110can include a steering wheel that is used in a first state to steer thevehicle and used in a second state as a controller for video gamecontent provided by the media system. Other example uses of the steeringwheel in the second state include navigating a menu (e.g., turn thesteering wheel to the right to scroll forward in the menu and turn thesteering wheel to the left to select an item in the menu), skippingforward or backward in a video or audio track (e.g., turn the steeringwheel to the right to skip forward and to the left to skip backward), orincreasing or decreasing the size of text or images displayed on thedisplay 205 (e.g., turn the steering wheel to the right to increase textfont size and to the left to decrease the font size). Other devices inthe vehicle 110 may likewise have different functions in differentstates, such as a gear shift, a brake pedal, or a turn signal lever.

In still other cases, the input devices 220 include one or moreinterfaces that receive a signal from a device that is not integratedinto the vehicle 110. For example, the input devices 220 can include aninterface configured to receive a signal from a mobile device orwearable device (such as a smart phone, smart watch, smart glasses, ortablet), such that the driver can provide inputs to the vehicle 110 viathe mobile or wearable device.

The vehicle 110 can have a comfort system 225. The comfort system 225can include one or more components with adjustable parameters thatfacilitate comfort of passengers in the vehicle 110. Example componentsof the comfort system include air temperature control components (suchas an air conditioning or heating system), air quality controlcomponents (such as a humidity or air purification system), seatwarmers, seat position controls, or chair massagers. In someembodiments, as shown in FIG. 2, the comfort system 225 can include oneor more subsystems to regulate comfort parameters for each seat 230 inthe vehicle. Thus, for example, the comfort system 225 can include afirst comfort system 225A corresponding to the driver seat 230A, asecond comfort system 225B corresponding to the passenger seat 230B, andso forth. In other embodiments, the comfort system is a centralizedcontrol system that defines parameters for comfort components throughoutthe vehicle, which may or may not be individualized to each seat 230.

FIG. 3 is a block diagram illustrating components of a vehicle 110,according to some embodiments.

As shown in FIG. 3, the vehicle 110 can include a vehicle experiencesystem 310. The vehicle experience system 310 controls an experience forpassengers in the vehicle 110. The vehicle experience system 310 caninclude computer software and hardware to execute the software,special-purpose hardware, or other components to implement thefunctionality of the media system 120 described herein. For example, thevehicle experience system 310 can include programmable circuitry (e.g.,one or more microprocessors), programmed with software and/or firmware,entirely in special-purpose hardwired (i.e., non-programmable)circuitry, or in a combination or such forms. Special-purpose circuitrycan be in the form of, for example, one or more application-specificintegrated circuits (ASICs), programmable logic devices (PLDs),field-programmable gate arrays (FPGAs), etc. In some embodiments, thevehicle experience system is implemented using hardware in the vehicle110 that also performs other functions of the vehicle. For example, thevehicle experience system can be implemented within an infotainmentsystem in the vehicle 110. In other embodiments, components such as oneor more processors or storage devices can be added to the vehicle 110,where some or all functionality of the vehicle experience system isimplemented on the added hardware.

The vehicle experience system 310 can read and write to a car network350. The car network 350, implemented for example as a controller areanetwork (CAN) bus inside the vehicle 110, enables communication betweencomponents of the vehicle, including electrical systems associated withdriving the vehicle (such as engine control, anti-lock brake systems,parking assist systems, and cruise control) as well as electrical systemassociated with comfort or experience in the interior of the vehicle(such as temperature regulation, audio systems, chair position control,or window control). The vehicle experience system 310 can also read datafrom or write data to other data sources 355 or other data outputs 360,including one or more other on-board buses (such as a local interconnectnetwork (LIN) bus or comfort-CAN bus), a removable or fixed storagedevice (such as a USB memory stick), or a remote storage device thatcommunicates with the vehicle experience system over a wired or wirelessnetwork.

The CAN bus 350 or other data sources 355 provide raw data from sensorsinside or outside the vehicle, such as the sensors 215. Example types ofdata that can be made available to the vehicle experience system 310over the CAN bus 350 include vehicle speed, acceleration, lane position,steering angle, in-cabin decibel level, audio volume level, currentinformation displayed by a multimedia interface in the vehicle, forceapplied by the user to the multimedia interface, ambient light, orhumidity level. Data types that may be available from other data sources355 include raw video feed (whether from sources internal or external tothe vehicle), audio input, user metadata, user state, calendar data,user observational data, contextual external data, traffic conditions,weather conditions, in-cabin occupancy information, road conditions,user drive style, or non-contact biofeedback. Any of a variety of othertypes of data may be available to the vehicle experience system 310.

Some embodiments of the vehicle experience system 310 process andgenerate all data for controlling systems and parameters of the vehicle110, such that no processing is done remotely (e.g., by the remoteserver 120). Other embodiments of the vehicle experience system 310 areconfigured as a layer interfacing between hardware components of thevehicle 110 and the remote server 120, transmitting raw data from thecar network 350 to the remote server 120 for processing and controllingsystems of the vehicle 110 based on the processing by the remote server120. Still other embodiments of the vehicle experience system 310 canperform some processing and analysis of data while sending other data tothe remote server 120 for processing. For example, the vehicleexperience system 310 can process raw data received over the CAN bus 350to generate intermediate data, which may be anonymized to protectprivacy of the vehicle's passengers. The intermediate data can betransmitted to and processed by the remote server 120 to generate aparameter for controlling the vehicle 110. The vehicle experience system310 can in turn control the vehicle based on the parameter generated bythe remote server 120. As another example, the vehicle experience system310 can process some types of raw or intermediate data, while sendingother types of raw or intermediate data to the server 120 for analysis.

Some embodiments of the vehicle experience system 310 can include anapplication programming interface (API) enabling remote computingdevices, such as the remote server 120, to send data to or receive datafrom the vehicle 110. The API can include software configured tointerface between a remote computing device and various components ofthe vehicle 110. For example, the API of the vehicle experience system310 can receive an instruction to apply a parameter to the vehicle froma remote device, such as a parameter associated with entertainmentcontent, and apply the parameter to the vehicle.

As shown in FIG. 3, some embodiments of the vehicle experience system310 can include a sensor abstraction component 312, an output module314, a connectivity adapter 316 a-b, a user profile module 318, asettings module 320, a security layer 322, an over the air (OTA) updatemodule 324, a processing engine 330, a sensor fusion module 326, and amachine learning adaptation module 328. Other embodiments of the vehicleexperience system 310 can include additional, fewer, or differentcomponents, or can distribute functionality differently between thecomponents. The components of the vehicle experience system 310 caninclude any combination of software and hardware, including, forexample, programmable circuitry (e.g., one or more microprocessors),programmed with software and/or firmware, entirely in special-purposehardwired (i.e., non-programmable) circuitry, or in a combination orsuch forms. Special-purpose circuitry can be in the form of, forexample, one or more application-specific integrated circuits (ASICs),programmable logic devices (PLDs), field-programmable gate arrays(FPGAs), etc. In some cases, the vehicle experience system 310 includesone or more processors, such as a central processing unit (CPU),graphical processing unit (GPU), or neural processing unit (NPU), thatexecutes instructions stored in a non-transitory computer readablestorage medium, such as a memory.

The sensor abstraction component 312 receives raw sensor data from thecar network 350 and/or other data sources 355 and normalizes the inputsfor processing by the processing engine 330. The sensor abstractioncomponent 312 may be adaptable to multiple vehicle models and can bereadily updated as new sensors are made available.

The output module 314 generates output signals and sends the signals tothe car network 365 or other data sources 360 to control electricalcomponents of the vehicle. The output module 314 can receive a state ofthe vehicle and determine an output to control at least one component ofthe vehicle to change the state. In some embodiments, the output module314 includes a rules engine that applies one or more rules to thevehicle state and determines, based on the rules, one or more outputs tochange the vehicle state. For example, if the vehicle state isdrowsiness of the driver, the rules may cause the output module togenerate output signals to reduce the temperature in the vehicle, changethe radio to a predefined energetic station, and increase the volume ofthe radio.

The connectivity adapter 316 a-b enables communication between thevehicle experience system 310 and external storage devices or processingsystems. The connectivity adapter 316 a-b can enable the vehicleexperience system 310 to be updated remotely to provide improvedcapability and to help improve the vehicle state detection modelsapplied by the processing engine. The connectivity adapter 316 a-b canalso enable the vehicle experience system 310 to output vehicle or userdata to a remote storage device or processing system. For example, thevehicle or user data can be output to allow a system to analyze forinsights or monetization opportunities from the vehicle population. Insome embodiments, the connectivity adapter can interface between thevehicle experience system 310 and wireless network capabilities in thevehicle. Data transmission to or from the connectivity adapter can berestricted by rules, such as limits on specific hours of the day whendata can be transmitted or maximum data transfer size. The connectivityadapter may also include multi-modal support for different wirelessmethods (e.g., 5G or WiFi).

The user profile module 318 manages profile data of a user of thevehicle (such as a driver). Because the automotive experience generatedby the vehicle experience system 310 can be highly personalized for eachindividual user in some implementations, the user profile modulegenerates and maintains a unique profile for the user. The user profilemodule can encrypt the profile data for storage. The data stored by theuser profile module may not be accessible over the air. In someembodiments, the user profile module maintains a profile for any regulardriver of a car, and may additionally maintain a profile for a passengerof the car (such as a front seat passenger). In other embodiments, theuser profile module 318 accesses a user profile, for example from theremote server 120, when a user enters the vehicle 110.

The settings module 320 improves the flexibility of systemcustomizations that enable the vehicle experience system 310 to beimplemented on a variety of vehicle platforms. The settings module canstore configuration settings that streamline client integration,reducing an amount of time to implement the system in a new vehicle. Theconfiguration settings also can be used to update the vehicle during itslifecycle, to help improve with new technology, or keep current with anygovernment regulations or standards that change after vehicleproduction. The configuration settings stored by the settings module canbe allowed locally through a dealership update or remotely using aremote campaign management program to update vehicles over the air.

The security layer 322 manages data security for the vehicle experiencesystem 310. In some embodiments, the security layer encrypts data forstorage locally on the vehicle and when sent over the air to determalicious attempts to extract private information. Individualanonymization and obscuration can be implemented to separate personaldetails as needed. The security and privacy policies employed by thesecurity layer can be configurable to update the vehicle experiencesystem 310 for compliance with changing government or industryregulations.

In some embodiments, the security layer 322 implements a privacy policy.The privacy policy can include rules specifying types of data that canor cannot be transmitted to the remote server 120 for processing. Forexample, the privacy policy may include a rule specifying that all datais to be processed locally, or a rule specifying that some types ofintermediate data scrubbed of personally identifiable information can betransmitted to the remote server 120. The privacy policy can, in someimplementations, be configured by an owner of the vehicle 110. Forexample, the owner can select a high privacy level (where all data isprocessed locally), a low privacy level with enhanced functionality(where data is processed at the remote server 120), or one or moreintermediate privacy levels (where some data is processed locally andsome is processed remotely).

Alternatively, the privacy policy can be associated with one or moreprivacy profiles defined for the vehicle 110, a passenger in thevehicle, or a combination of passengers in the vehicle, where eachprivacy profile can include different rules. In some implementations,where for example a passenger is associated with a profile that isported to different vehicles or environment, the passenger's profile canspecify the privacy rules that are applied dynamically by the securitylayer 322 when the passenger is in the vehicle 110 or environment. Whenthe passenger exits the vehicle and a new passenger enters, the securitylayer 322 retrieves and applies the privacy policy of the new passenger.

The rules in the privacy policy can specify different privacy levelsthat apply under different conditions. For example, a privacy policy caninclude a low privacy level that applies when a passenger is alone in avehicle and a high privacy level that applies when the passenger is notalone in the vehicle. Similarly, a privacy policy can include a highprivacy level that applies if the passenger is in the vehicle with adesignated other person (such as a child, boss, or client) and a lowprivacy level that applies if the passenger is in the vehicle with anyperson other than the designated person. The rules in the privacypolicy, including the privacy levels and when they apply, may beconfigurable by the associated passenger. In some cases, the vehicleexperience system 310 can automatically generate the rules based onanalysis of the passenger's habits, such as by using pattern tracking toidentify that the passenger changes the privacy level when in a vehiclewith a designated other person.

The OTA update module 324 enables remote updates to the vehicleexperience system 310. In some embodiments, the vehicle experiencesystem 310 can be updated in at least two ways. One method is aconfiguration file update that adjusts system parameters and rules. Thesecond method is to replace some or all of firmware associated with thesystem to update the software as a modular component to host vehicledevice.

The processing engine 330 processes sensor data and determines a stateof the vehicle. The vehicle state can include any information about thevehicle itself, the driver, or a passenger in the vehicle. For example,the state can include an emotion of the driver, an emotion of thepassenger, or a safety concern (e.g., due to road or traffic conditions,the driver's attentiveness or emotion, or other factors). As shown inFIG. 1, the processing engine can include a sensor fusion module, apersonalized data processing module, and a machine learning adaptationmodule.

The sensor fusion module 326 receives normalized sensor inputs from thesensor abstraction component 312 and performs pre-processing on thenormalized data. This pre-processing can include, for example,performing data alignment or filtering the sensor data. Depending on thetype of data, the pre-processing can include more sophisticatedprocessing and analysis of the data. For example, the sensor fusionmodule 326 may generate a spectrum analysis of voice data received via amicrophone in the vehicle (e.g., by performing a Fourier transform),determining frequency components in the voice data and coefficients thatindicate respective magnitudes of the detected frequencies. As anotherexample, the sensor fusion module may perform image recognitionprocesses on camera data to, for example, determine the position of thedriver's head with respect to the vehicle or to analyze an expression onthe driver's face.

The personalized data processing module 330 applies a model to thesensor data to determine the state of the vehicle. The model can includeany of a variety of classifiers, neural networks, or other machinelearning or statistical models enabling the personalized data processingmodule to determine the vehicle's state based on the sensor data. Oncethe vehicle state has been determined, the personalized data processingmodule can apply one or more models to select vehicle outputs to changethe state of the vehicle. For example, the models can map the vehiclestate to one or more outputs that, when effected, will cause the vehiclestate to change in a desired manner.

The machine learning adaptation module 328 continuously learns about theuser of the vehicle as more data is ingested over time. The machinelearning adaptation module may receive feedback indicating the user'sresponse to the vehicle experience system 310 outputs and use thefeedback to continuously improve the models applied by the personalizeddata processing module. For example, the machine learning adaptationmodule 328 may continuously receive determinations of the vehicle state.The machine learning adaptation module can use changes in the determinedvehicle state, along with indications of the vehicle experience system310 outputs, as training data to continuously train the models appliedby the personalized data processing module.

FIG. 4 is an example abstraction illustrating relationships betweencomponents and data associated with the vehicle 110. As shown in FIG. 4,the vehicle 110 can be represented as a series of layers including avehicle hardware sensor layer 410, a raw sensor data layer 420, a sensorfusion layer 430, an algorithm layer 440, a human-machine interfaceplatform layer 450, and an adaptive/responsive user experience layer460. The base layer of the vehicle abstraction includes vehicle sensors410, which are hardware components that output analog or digital signalsrepresenting parameters of the vehicle 110, a context of the vehicle110, or a state of a passenger in the vehicle. The sensor data layer 420represents initial processing of the signals into data types or signalsthat are usable by various processing components of the vehicle 110 orremote server 120. For example, the sensor data layer 420 representsnormalized or filtered data corresponding to the raw signals receivedfrom the sensors 215. The sensor fusion layer 430 represents a fusion ofthe sensor data 420, or an aggregation of the sensor data associatedwith multiple sensors 215 and collective analysis of this aggregatedsensor data to generate parameters for personalization or control of thevehicle 110. The human-machine interface layer 450 representsinteractions between a passenger in the vehicle 110 and the componentsof the vehicle 110. These interactions can include, for example, anycontent that the vehicle outputs to the passenger (such as mediacontent, navigational content, or requests for instructions) and anyinputs by the passenger into the vehicle. The interactions in thehuman-machine interface layer 450 can further include parameters appliedby the vehicle 110 to components interacted with by the passenger, suchas comfort settings or safety parameters implemented automatically bythe vehicle 110. The experience layer 460 represents an overallexperience of the passenger in the vehicle 110, based on the collectiveinteractions represented by the human-machine interface layer 450.

FIG. 5 is a block diagram illustrating an example configuration of thevehicle experience system with respect to other components of thevehicle.

The infotainment system 502, along with vehicle sensors 504 and vehiclecontrols 506, can communicate with other electrical components of thevehicle over the car network 350. The vehicle sensors 504 can includesome or all of the sensors 215 described with respect to FIG. 2, or caninclude different sensors. The vehicle controls 506 can control variouscomponents of the vehicle. For example, the vehicle controls 506 caninclude the entertainment system 210, the comfort system 225, and/orsafety controls or systems in the vehicle. A vehicle data logger 508 maystore data read from the car network bus 350, for example for operationof the vehicle. In some embodiments, the infotainment system 502 canalso include a storage device 510, such as an SD card, to store datarelated to the infotainment system, such as audio logs, phone contacts,or favorite addresses for a navigation system. The infotainment system502 can include an automotive system 310 that can be utilized toincrease user experience in the vehicle.

Although FIG. 5 shows that the vehicle experience system 310 may beintegrated into the vehicle infotainment system in some cases, otherembodiments of the vehicle experience system 310 may be implementedusing standalone hardware. For example, one or more processors, storagedevices, or other computer hardware can be added to the vehicle andcommunicatively coupled to the vehicle network bus, where some or allfunctionality of the vehicle experience system 310 can be implemented onthe added hardware.

Emotion Detection

FIG. 6A is a flowchart illustrating a process to determine the driver'semotional state, and FIG. 6B illustrates example data types detected andgenerated during the process shown in FIG. 6A. The processing in FIG. 6Ais described, by way of example, as being performed by the vehicleexperience system 310. However, in other embodiments, other componentsof the vehicle 110 or the remote server 120 can perform some or all ofthe analysis shown in FIG. 6A.

As shown in FIG. 6A, the vehicle experience system 310 can receive, atstep 602, data from multiple sensors associated with an automotivevehicle. In addition to the sensor data, the vehicle experience system310 may receive environmental data indicating, for example, weather ortraffic conditions measured by systems other than the vehicle experiencesystem 310 or the sensors associated with the vehicle. FIG. 6B shows, byway of example, four types of sensor data and two types of environmentaldata that can be received at step 602. However, additional or fewer datastreams can be received by the vehicle experience system 310.

As shown in FIG. 6B, the types of environmental data can include inputdata 610, emotional indicators 612, contextualized emotional indicators614, and contextual emotional assessment 616. The input data 610 caninclude environmental data 610 a-b and sensor data 610 c-f. Theemotional indicators 612 can include indicators 612 a-c. The contextualemotional indicators 614 can include indicators 614 a-c. In some cases,the contextual emotional indicators 614 a-c can be modified based onhistorical data 618. The contextualized emotional assessments 616 caninclude various emotional assessments and responses 616 a-b.

The vehicle experience system 310 generates, at step 604, one or moreprimitive emotional indications based on the received sensor (andoptionally environmental) data. The primitive emotional indications maybe generated by applying a set of rules to the received data. Whenapplied, each rule can cause the vehicle experience system 310 todetermine that a primitive emotional indication exists if a criterionassociated with the rule is satisfied by the sensor data. Each rule maybe satisfied by data from a single sensor or by data from multiplesensors.

As an example of generating a primitive emotional indication based ondata from a single sensor, a primitive emotional indication determinedat step 604 may be a classification of a timbre of the driver's voiceinto soprano, mezzo, alto, tenor, or bass. To determine the timbre, thevehicle experience system 310 can analyze the frequency content of voicedata received from a microphone in the vehicle. For example, the vehicleexperience system 310 can generate a spectrum analysis identify variousfrequency components in the voice data. A rule can classify the voice assoprano if the frequency data satisfies a first condition or set ofconditions, such as having certain specified frequencies represented inthe voice data or having at least threshold magnitudes at specifiedfrequencies. The rule can classify the voice as mezzo, alto, tenor, orbass if the voice data instead satisfies a set of conditionsrespectively associated with each category.

As an example of generating a primitive emotional indication based ondata from multiple sensors, a primitive emotional indication determinedat step 604 may be a body position of the driver. The body position canbe determined based on data received from a camera and one or moreweight sensors in the driver's seat. For example, the driver can bedetermined to be sitting up straight if the camera data indicates thatthe driver's head is at a certain vertical position and the weightsensor data indicates that the driver's weight is approximately centeredand evenly distributed on the seat. The driver can instead be determinedto be slouching based on the same weight sensor data, but with cameradata indicating that the driver's head is at a lower vertical position.

The vehicle experience system 310 may determine the primitive emotionalindications in manners other than by the application of the set ofrules. For example, the vehicle experience system 310 may apply thesensor and/or environmental data to one or more trained models, such asa classifier that outputs the indications based on the data from one ormore sensors or external data sources. Each model may take all sensordata and environmental data as inputs to determine the primitiveemotional indications or may take a subset of the data streams. Forexample, the vehicle experience system 310 may apply a different modelfor determining each of several types of primitive emotionalindications, where each model may receive data from one or more sensorsor external sources.

Example primitive emotional indicators that may be generated by themedia selection module 220, as well as the sensor data used by themodule to generate the indicators, are as follows:

Primitive Emotional Indicator Description Sensor Needed Voice TimbreUnique Overtones and Microphone frequency of the voice. Categorized as:Soprano Mezzo Alto Tenor Bass Decibel Absolute decibel level of thehuman Microphone Level voice detected. Pace The cadence at which thesubject Microphone isspeaking Facial Anger The detection that theoccupant is Front Facing angry and unhappy with something Camera DisgustThe response from a subject of Front Facing distaste or displeasureCamera Happiness Happy and general reaction of Front Facing pleasureCamera Sadness Unhappy or sad response Front Facing Camera SurpriseUnexpected situation Front Facing Camera Neutral No specific emotionalFront Facing response. Camera Body Force of The level of pressureapplied to the Entertainment/ Touch Entertainment screen with a userInfotainment screen interaction Body The position of the subject body,Camera + Occupant Position detectedbycomputer vision in Weight Sensorcombination with the seat sensors and captured in X, Y, Z coordinates

Based on the primitive emotional indications (and optionally also basedon the sensor data, the environmental data, or historical dataassociated with the user), the vehicle experience system 310 generates,at step 606, contextualized emotional indications. Each contextualizedemotional indication can be generated based on multiple types of data,such as one or more primitive emotional indications, one or more typesof raw sensor or environmental data, or one or more pieces of historicaldata. By basing the contextualized emotional indications on multipletypes of data, the vehicle experience system 310 can more accuratelyidentify the driver's emotional state and, in some cases, the reason forthe emotional state.

In some embodiments, the contextualized emotional indications can bedetermined by applying a set of rules to the primitive indications. Forexample, the vehicle experience system 310 may determine that contextualemotional indication 2 shown in FIG. 6B exists if the system detectedprimitive emotional indications 1, 2, and 3. Below is an exampleemotional indication model including rules that can be applied by thevehicle experience system 310:

Happy:

-   -   Event Detected: Mouth changes shape, corners turn upwards,        timbre of voice moves up half an octave    -   Classification: Smile    -   Contextualization: Weather is good, traffic eases up    -   Verification: Positive valence    -   Output/Action: Driver is happy, system proposes choices to        driver based on ambience, music, driving style, climate control,        follow-up activities, linked activities, driving route,        suggestions, alternative appointment planning, continuous        self-learning, seat position, creating individualized routines        for relaxation or destressing.

In other cases, the contextualized emotional indications can bedetermined by applying a trained model, such as a neural network orclassifier, to multiple types of data. For example, primitive emotionalindication 1 shown in FIG. 6A may be a determination that the driver ishappy. The vehicle experience system 310 can generate contextualizedemotional indication 1—a determination that the driver is happy becausethe weather is good and traffic is light—by applying primitive emotionalindication 1 and environmental data (such as weather and traffic data)to a classifier. The classifier can be trained based on historical data,indicating for example that the driver tends to be happy when theweather is good and traffic is light, versus being angry, frustrated, orsad when it is raining or traffic is heavy. In some cases, the model istrained using explicit feedback provided by the passenger. For example,if the vehicle experience system 310 determines based on sensor datathat a person is stressed, the vehicle experience system 310 may ask theperson “You appear to be stressed; is that true?” The person's answer tothe question can be used as an affirmative label to retrain and improvethe model for better determination of the contextualized emotionalindications.

The contextualized emotional indications can include a determination ofa reason causing the driver to exhibit the primitive emotionalindications. For example, different contextualized emotional indicationscan be generated at a different times based on the same primitiveemotional indication with different environmental and/or historicaldata. For example, as discussed above, the vehicle experience system 310may identify a primitive emotional indication of happiness and a firstcontextualized emotional indication indicating that the driver is happybecause the weather is good and traffic is light. At a different time,the vehicle experience system 310 may identify a second contextualizedemotional indication based on the same primitive emotional indication(happiness), which indicates that the driver is happy in spite of badweather or heavy traffic as a result of the music that is playing in thevehicle. In this case, the second contextualized emotional indicationmay be a determination that the driver is happy because she enjoys themusic.

Finally, at step 608, the vehicle experience system 310 can use thecontextualized emotional indications to generate or recommend one ormore emotional assessment and response plans. The emotional assessmentand response plans may be designed to enhance the driver's currentemotional state (as indicated by one or more contextualized emotionalindications), mitigate the emotional state, or change the emotionalstate. For example, if the contextualized emotional indication indicatesthat the driver is happy because she enjoys the music that is playing inthe vehicle, the vehicle experience system 310 can select additionalsongs similar to the song that the driver enjoyed to ensure that thedriver remains happy. As another example, if the driver is currentlyfrustrated due to heavy traffic but the vehicle experience system 310has determined (based on historical data) that the driver will becomehappier if certain music is played, the vehicle experience system 310can play this music to change the driver's emotional state fromfrustration to happiness. Below are example scenarios and correspondingcorrective responses that can be generated by the vehicle experiencesystem 310:

Contex- Primitive tualized Emotional Personalized Emotional Indicatorsand Corrective Scenario Description Sensors Response Safety Road Thepersonalized Vehicle Audio and Rage assessment that a Power Train -visual warning driver is aggravated Speed, for driver to be to the pointthat Acceleration aware of their actions could External - situation.harm themselves or Traffic, Massage others. This weather activated onassessment will take Emotional seat. into consideration Indicators ofTemperature the history of the Anger and reduced in specific user andDisgust vehicle have a Body Position Mood lighting personalized Deltas -adjusted to be threshold it will learn Physical less upsetting (noovertime Agitation red) Enter- tainment Head The physical Front FacingNone - Captured Bop reaction a subject Camera (Facial Data Point to behas while listening to changes, used for analysis a media source. Thismouthing words) or joined with goes beyond simple Body other data forenjoyment, to the Position behavioral mode of physical (Delta) analysisand/or reaction the user Cabin monetization demonstrates. ThisMicrophone purposes can be (musicbpm, key parameterized as signature)Metal, Sway, Pop. Entertainment Media Metadata (song, artist, timestamp,volume change) Comfort Emotional This feature will Front Facing Changeof Stability assess the desired Camera Audio Station emotional state ofBody Massage the occupant, and Position activated on adjust the (Delta)seat. environment to Cabin Cabin maintain that state Microphonetemperature for the subject. The Infotainment adjusted in requestedstate will Status vehicle be requested by the Mood lighting user, andcan be adjustments Calm, Sad, Intense or Seat Happy. temperature

The following table illustrates other example state changes that can beachieved by the vehicle experience system 310, including the data inputsused to determine a current state, an interpretation of the data, andoutputs that can be generated to change the state.

Emotional System Scenario Data Input Interpretation Output Stress DriverMonitoring Facial Coding Alternative Route Reduction Camera analysisSuggestions Analog Microphone Voice frequency Interactive spoken promptsSignal detection to driver DSP: Music beat Breathing Enhanced proactivedetection patterns communication regarding DSP: Processed audioDeviation from uncontrollable stress factors: CAN Data: Speed historicaluser weather, traffic conditions, CAN Data: Acceleration behaviorlocation of fueling and rest CAN Data: In-cabin Intensity of areas, etc.decibel level acceleration Activation of adaptive cruise External data:Traffic Anomaly control (ACC) conditions detection from norm Activationof Lane Assist External data: Weather Pupils dilated Modify the lightexperience condition Posture Air purification activated recognitionRegulate the sound level Gesture detection Aromatherapy activationRestlessness Dynamic audio volume detection modification Dynamic drivemode adjustment Activate seat massage Adjust seat position Music DriverMonitoring Posture Lighting becomes Enjoyment Camera recognitiondynamically reactive to music Analog Microphone Gesture detection Alldriver assist functions Signal Voice frequency activated (e.g. ACC, LaneDSP: Music beat detection Assist) detection Facial expressionDynamically-generated DSP: In-Cabin Decibel change music recommendationsLevel Zonal designed for the specific length CAN Data: Humiditydetermination of of the journey Detection music enjoyment Deactivateseat massage CAN Data: Acceleration Facial Expression Lower temperaturebased CAN Data: Increase in determination on increased movement andvolume level Voice frequency humidity CAN Data: Audio screen detectionDynamic drive mode in MMI Upper body pose adjustment to comfort modeExternal data: Traffic estimation When car stopped, Karaoke conditionsCorrelation to Mode activated External data: Weather past user behaviorconditions Detect audio key signature Intensity of acceleration RoadRage External data: Traffic Upper body pose Alternative Route Abatementconditions Facial Expression Suggestions Driver Monitoring determinationInteractive spoken prompts Camera Voice frequency to driver AnalogMicrophone detection Explain through simple Signal Breathing languagethe contributing DSP: Processed Audio patterns stress factors: EnhancedSignal Deviation from proactive communication CAN Data: Audio historicaluser regarding uncontrollable stress volume level behavior factors:weather, traffic CAN Data: Distance to Intensity of conditions, locationof fueling car ahead acceleration and rest areas, etc. CAN Data: LaneCheck for erratic Activation of ACC position driving Activation of LaneAssist CAN Data: Speed Anomaly Modify the light experience CAN Data:Acceleration detection from norm Regulate the sound level CAN Data:In-cabin Pupils dilated Air purification activated decibel level PostureAromatherapy activation External data: Weather recognition Dynamic audiovolume conditions Gesture detection modification DSP: External noiseRestlessness Dynamic drive mode pollution detection adjustment CAN Data:Force touch Steering style Activate seat massage detection RestlessnessAdjust seat position CAN Data: Steering Adjust to average user wheelangle comfort setting Body seating position CAN Data: Passenger seatinglocation Tech Detox Driver Monitoring Facial stress CountermeasuresCamera detection Scent Analog Microphone Voice frequency Music Signalchanges Alternative Route Cobalt DSP: In-Cabin Breathing SuggestionsDecibel Level patterns Spoken CAN Data: Ambient Slow response toAcceleration- Air purification Light Sensor factors activated CAN Data:Infotainment Pupils dilated Activation of security system Force TouchPosture Proactive communication CAN Data: Decrease in recognitionregarding weather, traffic volume level Gesture detection conditions,rest areas, etc. CAN Data: Audio screen Color Dynamic drive mode ininfotainment system temperature of in- adjustment External data: Trafficvehicle lights conditions Correlation with External data: Weatherweather condition Do Not CAN Data: Passenger Rate of changeCountermeasures Disturb seating against expected Scent location normMusic Body seating position Frequency Alternative Route DriverMonitoring Intensity Suggestion Camera Delta of detected Spoken AnalogMicrophone events from typical Acceleration Signal status Activation ofsecurity system DSP: Processed Audio Proactive communication Signalregarding weather, traffic CAN Data: Audio conditions, rest areas, etc.volume level Dynamic drive mode CAN Data: Drive mode: adjustment comfortCAN Data: In-cabin decibel level CAN Data: Day and Time External data:Weather conditions DSP: External noise pollution Drowsiness DriverMonitoring Zonal detection Countermeasures Camera Blink detection ScentBody seating position Drive Style Music Analog Microphone Steering StyleAlternative Route Signal Rate of change Suggestions DSP: Processed Audioagainst expected Spoken Signal norm Acceleration - Air purification CANData: Steering Frequency activated Angle Intensity Activation ofsecurity CAN Data: Lane Delta of detected systems departure event fromtypical Proactive communication CAN Data: Duration of status regardingweather, traffic journey conditions, rest areas, etc. CAN Data: Day and“Shall I open the windows?” Time Dynamic drive mode CAN Data: RoadProfile adjustment Estimation Significant cooling of interior Externaldata: Weather cabin temperature conditions Adapting driving mode to CANData: Audio level auto mode (detect the bumpy road) Driver DriverMonitoring Rate of change Countermeasures Distraction Camera againstexpected Scent Analog Microphone norm Music Signal Frequency AlternativeRoute DSP: Vocal frequency Intensity Suggestions DSP: Processed audioDelta of detected Spoken CAN Data: Lane events from typical Accelerationdeparture status Activation of security CAN Data: MMI Force systemsTouch Proactive communication CAN Data: In-cabin regarding weather,traffic decibel level conditions, rest areas, etc. CAN Data: MobileDynamic drive mode phone notification and call adjustment informationExternal data: Traffic conditions External data: Weather condition

Current implementations of emotion technology suffer by their relianceon a classical model of Darwinian emotion measurement andclassification. One example of this is the wide number of facialcoding-only offerings, as facial coding on its own is not necessarily anaccurate representation of emotional state. In the facial coding-onlymodel, emotional classification is contingent upon a correlationalrelationship between the expression and the emotion it represents (forexample: a smile always means happy). However, emotions are typicallymore complex. For example, a driver who is frustrated as a result ofheavy traffic may smile or laugh when another vehicle cuts in front ofhim as an expression of his anger, rather than an expression ofhappiness. Embodiments of the vehicle experience system 310 take acausation-based approach to biofeedback by contextualizing each datapoint that paints a more robust view of emotion. These contextualizedemotions enable the vehicle experience system 310 to more accuratelyidentify the driver's actual, potentially complex emotional state, andin turn to better control outputs of the vehicle to mitigate or enhancethat state.

FIG. 7 is a flowchart illustrating another process 700 for detecting anemotional state of a person, according to some embodiments. The process700 can be performed by the vehicle experience system 310, although theprocess 700 is not limited to execution in a vehicle. The process 700can represent a person's emotional state as a comparison to anotheremotional state. Thus, the emotional state generated by the vehicleexperience system 310 using the process 700 may not include an explicitdetermination, for example, that a driver is stressed, but rather thatthe driver is exhibiting emotional indications different from thoseexhibited in the driver's neutral state and that cause the vehicle 110to implement a stress mitigation response. Other embodiments of theprocess 700 can include additional, fewer, or different steps than thoseshown in FIG. 7, for example to include one or more of the stepsdescribed with respect to FIG. 6A.

As shown in FIG. 7, the vehicle experience system 310 detects, at step702, a preliminary emotional state of a person. The preliminaryemotional state can, in some cases, be an emotional state measurednon-contextually at a first time. In other cases, the preliminaryemotional state can be a baseline emotional state. The baselineemotional state can be determined based on data received from multiplesensors in the vehicle 110, each of which is configured to measure adifferent parameter of the person. The baseline emotional state canrepresent one or more primitive emotional indications that aredetermined to correspond to a neutral state of the passenger. The“neutral” state can be determined, for example, based on an amount oftime the passenger exhibits the primitive emotional indications, suchthat a primitive emotional indication exhibited for a greatest amount oftime is identified as an indication of the neutral state. Alternatively,the neutral state can be determined by identifying a time the passengeris expected to be in a neutral state, such as a time when traffic andweather are moderate. The primitive emotional indications can begenerated as described with respect to FIG. 6A.

At step 704, the vehicle experience system 310 detects a change in theperson's emotional state based on the data received from the sensors215. For example, the vehicle experience system 310 detects one or moreprimitive emotional indications that are different than the primitiveemotional indications associated with the preliminary emotional state.The detected change can, by way of example, be represented as acontextual emotional indication.

Based on the detected change in the person's emotional state, thevehicle experience system 310 controls a parameter in an environment ofthe person. For example, the vehicle experience system 310 can controlone or more parameters of an entertainment, comfort, or safety system inthe vehicle 110 based on the changed emotional state.

Personalization Overview

As noted above, various characteristics of an environment (e.g., adetected emotion of a vehicle passenger) can be obtained and one or moreoutput actions can be performed based on the obtained characteristics.For instance, various features of a vehicle can be modified (e.g., lightbrightness levels can be modified) based on detecting an emotional stateof a driver of a vehicle using a series of vehicle sensors. Further, theoutput actions can be based on user-specific information to provide apersonalized user experience in the vehicle.

FIG. 8 is an example flow diagram 800 of a process for generatinguser-specific output actions in a vehicle. As shown in FIG. 8, a processmay include acquiring various user characteristic information thatrelates to a particular user (block 802). Examples of usercharacteristic information can include biometric information of a user,previous feedback information provided by the user, driving patterns andtendencies by the user, preferred vehicle settings for the user, etc.The user characteristic information can be acquired and processed todevelop a profile of the user that can be utilized as a model ingenerating user-specific output actions.

The process can include processing user characteristic information togenerate a user profile (block 804). For example, driving tendencies andpatterns of the user can be processed to determine a driving profiletype that corresponds to one of a series of predetermined drivingprofiles. As another example, the profile can include vehicle settingsthat are specific to the user. The profile of the user can be used todevelop a model that correlates various vehicle inputs and correspondingvehicle output actions.

The process can include identifying user(s) in the vehicle (block 806).For instance, facial recognition can be used to identify a driver in adriver region of the vehicle. As another example, other biometricinformation (e.g., height/weight of the user) can be acquired toidentify a passenger in a passenger region of the vehicle. Uponidentifying the user in a region of the vehicle (e.g., driver seat,passenger seat), user characteristic information and/or a user-specificmodel associated with the identified user can be retrieved and/orderived.

The process can include detecting an emotion or emotional state relatingto the user (block 808). The emotional state of the user may impact theresultant output action in the vehicle. Detecting an emotional state ofthe user may include any of the features as discussed above. Asdescribed in greater detail below, features relating to the emotionalstate of a user can be user-specific.

The process may include generating user-specific output actions in thevehicle (block 810). The output actions can include actions performed bycomponents of the vehicle either interior of the vehicle or exterior ofthe vehicle. For instance, an output action can include playing aspecific form of media (e.g., audio/video) in the vehicle, adjusting thetemperature in the vehicle, modifying light settings in the vehicle,changing computer-generated directions to a destination, etc. Theuser-specific output actions can be based on any of the user profile forthe user and the detected emotional state of the user.

As an example, a series of user characteristics (e.g., a preferredcoffee order at a coffee shop, a preferred cuisine, payment information,and a destination location) can be retrieved relating to a user. Upondetecting an input event (e.g., detecting congestion on the path to aninitial destination), a user profile for the user can be processed todetermine that the user generally stops for coffee at the coffee shopwhen congestion on a road occurs. The system can also determine that anemotional state of the user is relaxed (e.g., the user does not have anymeetings scheduled, the time of day is generally when the user is notbusy). Responsive to determining that the user has accepted therecommended action of stopping at the coffee shop, a coffee order forthe user at the coffee shop can be placed and the destination can bemodified to include the location of the coffee shop. The present processmay provide user-specific output actions that can increase userexperience in the vehicle.

Example Method for Performing Output Actions Based on a User-SpecificProfile Model

In some embodiments, the present embodiments relate to generating auser-specific profile model and implementing an output action based onthe user-specific profile model. The user-specific profile model may beiteratively updated to account for new information relating touser-implemented modifications to the vehicle environment. Outputactions (e.g., modifications to features of the vehicle) can begenerated and implemented using the user-specific profile model.

The method may include retrieving a series of predetermined profiletypes. Each predetermined profile type may include a series of featurescommon among a subset of users. For example, a predetermined profiletype can include a “health/fitness” profile type that includes featurescommon among individuals that prioritize their personal health andfitness.

The method may include acquiring a stream of user characteristic dataindicative of modifications to settings of a vehicle implemented by theuser. For instance, the user characteristic data can include preferredinternal climate settings that is implemented by the user. As anotherexample, the user characteristic data can include a destinationrequested by the user at a specific time.

The method may include, for each portion of user characteristic data ofthe stream of user characteristic data, comparing the usercharacteristic data with the series of predetermined profile types todetermine a first predetermined profile type with features thatcorrespond to the user characteristic data.

In some embodiments, the method may include presenting an emotionalstate request to the user requesting an audio response indicative of anemotional state of the user, receiving the audio response indicative ofthe emotional state of the user, processing the audio response to derivethe emotional state of the user, wherein the emotional state isprocessed with the set of vehicle environmental data using theuser-specific profile model to generate an updated output action thatmodifies one or more vehicle features.

In some embodiments, the method may include identifying an identity ofthe user located within the vehicle. The user-specific profile model maybe retrieved responsive to identifying the identity of the user.Identifying the identity of the user may be based on any of comparingbiometric data included in the data acquired by the series of sensorswith a listing of biometric data of a series of known users andreceiving an identification message from a mobile device associated withthe user that is indicative of the identity of the user.

The method may include, for each portion of user characteristic data ofthe stream of user characteristic data, modifying a user-specificprofile model to include the determined features that correspond to theuser characteristic data. This may personalize the user-specific profilemodel, as a series of features that correspond to user characteristicdata are included in the user-specific profile model. The user-specificprofile model can be used to generate output actions that are unique tothe user.

The method may include processing a set of vehicle environmental dataacquired by a series of sensors disposed on the vehicle using themodified user-specific profile model to generate an output action thatmodifies one or more vehicle features.

The method may include inspecting data acquired by the series of sensorsdisposed on the vehicle to identify an emotional state corresponding tothe user.

In some embodiments, identifying the emotional state corresponding tothe user further comprises inspecting the data acquired by the series ofsensors to identify a deviation between a set of baseline biometriccharacteristics and a set of obtained biometric characteristics that isindicative of the emotional state of the user and comparing thedeviation between the set of baseline biometric characteristics and theset of obtained biometric characteristics with a series of predeterminedemotional states to identify a predetermined emotional state withfeatures that correspond to the deviation between the set of baselinebiometric characteristics and the set of obtained biometriccharacteristics.

In some embodiments, identifying the emotional state corresponding tothe user further comprises retrieving a series of predeterminedemotional states, each predetermined emotional state including aplurality of features indicative of an emotion, and comparing the dataacquired by the series of sensors with the series of predeterminedemotional states to identify a predetermined emotional state withfeatures that correspond to the data acquired by the series of sensors,wherein the identified emotional state includes features of thepredetermined emotional state.

In some embodiments, the method may include determining an operationalstate of the vehicle that includes an operating state and an idle state,wherein only a subset of output actions are allowed to be performed whenin the operating state, determining that the operational state of thevehicle has transitioned from the idle state to the operating state,responsive to determining that the operational state of the vehicle hastransitioned from the idle state to the operating state, presenting aseries of recommended output actions to the user that are capable ofbeing performed in the transitioned operational state, and performing anumber of the recommended output actions responsive to obtaining aresponse from the user indicating a confirmation to perform the numberof recommended output actions.

The method may include performing the generated output action to modifyone or more features of the vehicle. In some embodiments, the method mayinclude presenting a series of output actions capable of being performedon the vehicle to the user by processing the set of vehicleenvironmental data using the user-specific profile model, and receivingan indication to perform an identified output action of the series ofoutput actions presented to the user, wherein the identified outputaction is performed on the vehicle.

In some embodiments, the method may include presenting a request forfeedback to the user relating to the output actions performed on thevehicle, and receiving a response from the user including an indicationof whether the output action performed on the vehicle was appropriate,wherein the user-specific profile model is updated to include thevehicle environment characteristics and the generated output action.

In some embodiments, the method may include determining that a routeddestination of the vehicle is within a threshold distance of an externaldevice environment associated with the user, and responsive todetermining that the routed destination of the vehicle is within thethreshold distance of the external device environment associated withthe user, sending an indication to the external device environmentincluding a request to perform output actions that correspond to the oneor more output actions performed on the vehicle.

In some embodiments, the vehicle environment may be configured toperform steps relating to various medical applications. For instance,the series of sensors in the vehicle can acquire data relating to a userthat can be utilized in diagnosing various medical conditions. Asanother example, the infotainment system or heads up display caninitiate a video conference with a medical provider over an encryptedcommunication interface. The medical provider can access and review dataacquired by the sensors in the vehicle to diagnose various illnesses andprovide medical advice to the user in the vehicle.

Acquisition of User Characteristic Information

As described above, user characteristic information representing variousfeatures relating to the user can be acquired. A series of vehiclesensors can be utilized in capturing data that can be processed toderive user characteristic information.

One such example of user characteristic information can include drivingtendencies of the user. Driving tendencies can include common tendenciesof the user's driving style. For example, a type of road (e.g., dirtroad, highways, gravel) and a relative speed on each type of road may beincluded as driving tendencies. As another example, driving tendenciescan include an aggressiveness in turning the vehicle, an average speedof the vehicle at various times, average stopping times of the vehicle,an average number of passengers in the vehicle, common destinations inthe vehicle, etc.

Driving tendencies can be utilized in generating a driving profile forthe user, which is described in greater detail below. The drivingprofile can modify output actions for the user.

Another example of user characteristic information can includeinformation relating to the user derived from a series of sensors in thevehicle. Information relating to the user can include desired vehiclesettings (e.g., seat position, vehicle temperature, light brightnesssettings), user-specific emotional state characteristics (e.g., aheartbeat of the user, a gaze of the user), etc.

The user characteristic information can be retrieved from third-partysources. For instance, information gathered from social mediaapplications, a smart home environment, sensor data from apreviously-occupied vehicle by the user, etc.

In some embodiments, user characteristic information may be updatedbased on feedback provided by the user. For example, upon modifying atemperature of the vehicle, the system can prompt the user to indicatewhether the modified temperature is preferable to the user. The responseprovided by the user can be indicative of whether the user prefers themodified temperature or not.

The user characteristic information can include media settings, such asvideo/audio playback settings, volume controls, etc. For instance, if auser pauses a video playing on a display, the time position of the videowhen it was paused can be stored and associated with the user.Accordingly, when the user wants to continue playback of the video, thesystem can retrieve the time position of the video when it was pausedand resume play at that time position.

The user characteristic information can be formatted in a specificformat and maintained locally and/or remotely by computing resources inthe computing environment as described herein. For instance, database(s)can sort various user characteristic information types for the user suchthat the user characteristic information can be retrieved for furtherprocessing.

User Profile Generation

A user profile for a user can be generated based on the usercharacteristic information. A user profile can include a model that canbe utilized in personalizing output actions and recommended outputactions for a user in the vehicle. For instance, the user profile can beindicative of driving tendencies and/or desired vehicle settings for auser.

In some embodiments, a user can initially include a default userprofile. The default user profile can include a universal series ofcharacteristics that correspond to default settings for users. Forinstance, the default user profile can include a neutral set of settingsthat correspond to an average user of a vehicle. In many cases, thedefault user profile can be geographically-specific, i.e. the defaultuser profile can be modified based on a location of the vehicle. Forexample, a first default user profile in an urban location may includesettings that correspond to drivers of the vehicle commuting on highwaysand urban roads. Additionally, a second default profile in a rurallocation can include different settings that generally correspond todrivers of vehicles that operate the vehicles off-road or in variousdriving conditions. The default user profile can be iteratively modifiedto be a user-specific profile based on continuously obtaining moreinformation relating to the user.

Generating the user profile can include identifying one of a series ofprofile types that corresponds to the user characteristic information. Aprofile type in the series of profile types can include a set of commoncharacteristics that is common to a plurality of users. For example, aprofile type can include a “car enthusiast” type that includes commonlyassociated features relating to users that generally have positiveinteractions with performance features of the vehicle (e.g., drivingoff-road, driving at higher speeds). As another example, a profile typecan include an “executive” type that includes commonly associatedfeatures relating to users that generally have positive interactionswith business-related features of the vehicle (e.g., identifyingmeetings on a synchronized calendar, identifying routes to a destinationthat have a lowest drive time).

As more user characteristic information is gathered, pattern matchingmay be used to match characteristics in the user characteristicinformation with features included in the series of profile types. Forinstance, identifying that the user is driving the vehicle in congestedareas can match features of an “executive” profile type that relates tousers that commonly drive in congested areas. As another exampleidentifying that the user frequently enables power-saving settings inthe vehicle may match features with a “eco-friendly” profile type thatrelates to users that commonly attempt to energy-efficiency of thevehicle.

In some embodiments, a user profile can be generated that is unique tothe user. For instance, various aspects of each of the series of profiletypes can be incorporated into a new profile for the user. For example,as features are matched with a profile type, the matched feature of theprofile type can be incorporated into the new profile for the user.

FIG. 9 is an example block diagram 900 for generating a user profile. Asshown in FIG. 9, a user profile 902 can be based on both usercharacteristic information 904 and profile types 906 a-n. For instance,user characteristic information 902 can be compared with the profiletypes 906 a-n to match a specific feature included in the usercharacteristic information 902 with a corresponding profile types 906a-n. Example user characteristic information 904 features can includedriving tendencies 908 a, location information 908 b, emotional state908 c, biometric information 908 d, and any other information 908 n.

As shown in FIG. 9, a feature 908 a-n of the user characteristicinformation 904 can be compared with features 910 a-l of each profiletype 906 a-n to identify a profile type that corresponds to eachacquired feature 908 a-n. As an example, if a driving tendency 908 aindicates that the user operates the vehicle in off-road conditions orin a manual-transmission state, the user profile 902 can identify that acar enthusiast profile 906 a corresponds to this driving tendency 908 a.Identifying a profile type 906 a-n that corresponds to a portion of usercharacteristic information 904 can include extracting features 908 a-nof the portion of user characteristic information 904 and identifyingfeatures 910 a-l of each profile type 906 a-n to identify a profile typethat includes a number of similar features that exceeds a thresholdlevel or includes a greatest number of similar features of all profiletypes.

Upon identifying a profile type 906 a-n that corresponds to a portion ofuser characteristic information 904, the user profile 902 can retrievefeatures associated with the corresponding profile type and associatethose features with user in the user profile 902. As more usercharacteristic information 904 is acquired, the user profile 902 can beupdated with more features that correspond to various profile types 906a-n. In some embodiments, if the user profile 902 includes a number ofportions of user characteristic information 904 that corresponds to aprofile type that exceeds a threshold similarity, the user profile 902correlates all features of that profile type to the user profile 902. Inother embodiments, the user profile 902 includes features 910 a-l ofvarious profile types 906 a-n. This may result in the generation of aunique user profile 902 that is specific to each user.

A profile model can be generated that is indicative of user intereststhat can be used in user-specific info generation. A personalized dataprocessing module may apply a model to the sensor data to determine thestate of the vehicle. The model can include any of a variety ofclassifiers, neural networks, or other machine learning or statisticalmodels enabling the personalized data processing module to determine thevehicle's state based on the sensor data. Once the vehicle state hasbeen determined, the personalized data processing module can apply oneor more models to select vehicle outputs to change the state of thevehicle. For example, the models can map the vehicle state to one ormore outputs that, when effected, will cause the vehicle state to changein a desired manner.

The machine learning adaptation module can continuously learn about theuser of the vehicle as more data is ingested over time. The machinelearning adaptation module may receive feedback indicating the user'sresponse to the vehicle experience system outputs and use the feedbackto continuously improve the models applied by the personalized dataprocessing module. For example, the machine learning adaptation modulemay continuously receive determinations of the vehicle state. Themachine learning adaptation module can use changes in the determinedvehicle state, along with indications of the vehicle experience systemoutputs, as training data to continuously train the models applied bythe personalized data processing module.

User Detection and Identification

In some embodiments, one or more users in a vehicle can be detectedusing various techniques. Detecting on or more users in the vehicle mayallow for retrieval of a user profile of the user that can be used inpersonalizing output actions in the vehicle.

The set of sensors disposed in the vehicle can be used to detect one ormore users in the vehicle. As an example, various biometric informationassociated with a user can be used in detection of a user. For instance,a camera can capture facial features of a user, and facial recognitiontechniques can be implemented to identify a user based on the capturedfacial features of the user. Other examples of biometric informationthat can be used in user detection can include a weight of the user, aheight of the user, a fingerprint of the user, etc.

In some embodiments, detecting a user can include acquiring audiorepresenting a voice of a user and processing the audio to derive a userbased on their voice. Processing the voice of the user can includedetecting words, phrases, tones, etc. of the voice to identify a userfrom the voice. The voice of the user can be captured by a microphonedisposed in the vehicle.

In some embodiments, detecting a user can include the vehicle connectingto an electronic device associated with the user and identifying theuser based on an identifier of the electronic devices that is indicativeof the user. For instance, when a mobile phone associated with the useris within a communication range of a communication component (e.g.,antenna) of the vehicle, the mobile phone may provide an indicator thatidentifies the individual.

In some embodiments, a user can be detected responsive to obtaining apassword or passcode that is unique to the user. For example, this caninclude providing a password on an application of a mobile phoneassociated with the user or providing a password directly to thevehicle.

Upon detecting a user, the vehicle may provide a prompt requestingconfirmation of the identity of the user. Upon obtaining a confirmation(e.g., via voice command, a passcode) from the user, the user profilerelating to the user can be retrieved.

In some embodiments, multiple users can be individually identified invarious regions of a vehicle. For instance, a first user can be detectedin a driver seat region of a vehicle and a second user can be detectedin a passenger seat region of the vehicle. While various seats of avehicle are provided as examples herein, the present embodiments mayrelate to any suitable vehicle or environment that includes multipleusers.

FIG. 10 is a block diagram of an example vehicle 1000 with multipleregions. As shown in FIG. 10, a vehicle can include multiple regions(e.g., four regions). For instance, a first region 1002 a can include adriver seat region, and regions 1002 b-d can include passenger seatregions. The series of sensors disposed on the vehicle can identifyusers located within each region of the vehicle. A unique user can beidentified in each region of the vehicle, and a unique user profile foreach user can be retrieved and utilized in generating user-specificoutput actions that are unique to each region.

In some embodiments, the number of features and output actions availableto a user in the vehicle can be modified based on a region that the useris located. For instance, while in a first region 1002 a, a user may beunable to view video content while the vehicle is operating, while auser in a second region 1002 b can view video content without anyrestriction. As another example, a user in region 4 1002 d can modify abrightness level of lights directed toward region 4 1002 d, while a userin region 3 1002 c can uniquely modify a temperature of the areaincluded in region 3 1002 c.

In some embodiments, user profile information can be translated betweenvarious vehicles upon identifying the individual. For instance, upon auser entering a rental vehicle, the user profile of the user can beobtained by the rental vehicle. As another example, the user profile canbe integrated into a ride-sharing vehicle.

Emotional State Detection

As noted above, an emotional state of a user in the vehicle can bedetected. For instance, a series of sensors can inspect facial featuresand/or biometric information (e.g., a heartbeat) of a user to determinewhether the user is stressed or relaxed. Further, detection of anemotional state of a user can be user-specific. In other words,user-specific emotional information can be included in the user profilethat can be used in accurately detecting an emotional state of the userbased on characteristics that are unique to the user.

Any of the series of sensors disposed in the vehicle can be utilized indetecting an emotional state of the user. For instance, cameras cancapture images of the user, and the images can be processed to determinewhether facial features of the user indicate that the user is stressedor relaxed.

Further, emotional state information relating to the user can be storedand added to a user profile associated with the user. Particularly,specific features indicative of an emotional state of the user can beadded to the user profile that can be utilized in identifying subsequentemotional states of the user. For example, the series of sensors on thevehicle can capture a gaze or a heartbeat of the user. In this example,and of the captured gaze and the heartbeat can be processed to detect anemotional state of the user. For instance, if a heartbeat of the user isgreater than an average rate, the detected emotional state may be“stressed.” Further, as an example, various heartbeat information (e.g.,resting heartrate, elevated heart rate) can be captured and added to theuser profile.

As another example, upon detecting an emotional state of the user, thesystem may provide a prompt to the user requesting confirmation of thedetected emotional state. For instance, the prompt may include thequestion “We have detected that you are stressed, is that correct?” Uponreceipt of a response from the user indicating that the emotional stateis correct or incorrect, the particular features that led to thedetermination of the emotional state can be included as part of the userprofile. If the particular features were unique to the user that areindicative of the emotional state, the features can be added to the userprofile as being indicative of a particular emotional state.

In some embodiments, the emotional state of a user can be derived from adefault set of emotional state characteristics that identify features ofeach emotional state. As more emotional state information is derivedwith respect to a user, the default set of emotional statecharacteristics can be modified for the user and stored as part of theuser profile.

Generation of Output Actions

As noted above, any of a number of output actions can be performed basedon the user profile and a detected emotional state of the user. Anoutput action can include modification of any suitable feature in avehicle. Output actions can relate to components interior of the vehicleor exterior of the vehicle. Example components that can be modified byan output action can include interior lighting, audio, video, displaysettings, seat settings (e.g., heated seats, seat massaging), seatpositioning, etc. A central display (or “infotainment system”) can alsobe modified in the output action. For instance, a destination on amapping application on the infotainment system can be updated.

FIG. 11 is an example block diagram 1100 for generating personalizedoutput actions. As shown in FIG. 11, input information 1104 and adetected emotion 1106 can be obtained. The obtained information can beprocessed using the user profile 1102 to determine a user-specificoutput action 1108. Example output actions can relate to media settings1108 a, seat settings 1108 b, display settings 1108 c, vehicleenvironment settings 1108 d, etc.

As an example, input information 1104 can include destination on aninfotainment system and a set of default vehicle interior settings. Anexample detected emotion 1106 can include an emotion of the user and canbe utilized in modifying the output actions.

The user profile 1102 can include user-specific features and a profiletype. The profile type can include any profile as described herein. Theinput information and detected emotion data processed by the userprofile can be used to generate output actions.

In some embodiments, feedback information 1110 can be requested from theuser. For example, after adjusting the seat settings, feedback 1110 canbe provided by the user indicating whether the user confirms the outputaction. The feedback information can be fed back into the user profilethat can be used in subsequent generation of personalized outputactions.

In some embodiments, the vehicle environment can interact with otherenvironments. For instance, information can be transmitted between avehicle and devices included in a smart home environment. Particularly,output actions implemented in the vehicle environment can be translatedto another environment.

As an example, upon determining that an output action is to play a songto energize the user, an instruction to play the song on a set ofspeakers in a smart home environment can be played when the vehicle iswithin a threshold distance to the smart home environment. In thisexample, as the user transitions from the vehicle to the smart homeenvironment, the song can transition from the speakers of the vehicle tospeakers in the smart home environment. In some embodiments, usercharacteristic information can be transmitted between devices in variousembodiments. For instance, the vehicle can obtain user characteristicinformation acquired from devices included in a smart home environment.

Output Action Generation Examples

In a first example, a user profile can be associated with an “executive”profile type. The executive profile type can include characteristicsthat relate to an urban professional. For example, the executive profiletype can indicate that the user typically drives on urban roads andhighways in congested traffic and that the user has various meetings intheir schedule to accommodate.

In this example, upon detecting that an emotional state of the user isrelaxed, the system can generate output actions to energize the user.For instance, the output action can include navigating the user to themost-time efficient route to a destination or to present an indicationto charge the vehicle between scheduled appointments. Alternatively,detecting that an emotional state of the user is angry or frustrated,the system can generate output actions to relax the user. For instance,the output action can include modifying a destination to stop for acoffee break for the user or to turn on lane assist features on the carto assist with maintaining proper lane positioning of the vehicle torelax the user.

In a second example, a user profile can be associated with a “digitalnative” profile type. The digital native profile type can includecharacteristics that relate to a user that utilizes technology featuresin the vehicle.

In this example, upon detecting that an emotional state of the user isrelaxed, the system can generate output actions to energize the user.For instance, the output action can include playing back music with acorresponding dynamic light show in the vehicle or rerouting thedestination to a route with lower air quality levels. Alternatively,detecting that an emotional state of the user is angry or frustrated,the system can generate output actions to relax the user. For instance,the output action can include rerouting the destination to stop at abeach or ordering a meal for the user.

In a third example, a user profile can be associated with a“sustainability” profile type. The sustainability profile type caninclude characteristics that relate to a user that desires to reduceemissions from the vehicle and increases energy efficiency.

In this example, upon detecting that an emotional state of the user isrelaxed, the system can generate output actions to energize the user.For instance, the output action can include routing the user to the mostenergy-efficient route to a destination or presenting a summary ofenergy efficiency statistics to the user. Alternatively, detecting thatan emotional state of the user is angry or frustrated, the system cangenerate output actions to relax the user. For instance, the outputaction can include modifying internal climate control responsive to achange in weather or modifying a route to a destination to a scenicroute.

In a fourth example, a user profile can be associated with a“health/fitness” profile type. The health/fitness profile type caninclude characteristics that relate to a user that prioritizes healthand wellness-based traits.

In this example, upon detecting that an emotional state of the user isrelaxed, the system can generate output actions to energize the user.For instance, the output action can include routing the destination ofthe vehicle to a gym or directing the user to a parking location fartheraway from the destination to allow for extra exercise by the user.Alternatively, detecting that an emotional state of the user is angry orfrustrated, the system can generate output actions to relax the user.For instance, the output action can include routing the destination tothe vehicle to a new restaurant or initiating playback of a sportshighlight reel.

In a fifth example, a user profile can be associated with a “carenthusiast” profile type. The car enthusiast profile type can includecharacteristics that relate to a user that prioritizes performancecapabilities of the vehicle.

In this example, upon detecting that an emotional state of the user isrelaxed, the system can generate output actions to energize the user.For instance, the output action can include modifying a transmissiontechnique to a sport mode or modifying the destination of the vehicle toa route that optimizes acceleration of the vehicle. Alternatively,detecting that an emotional state of the user is angry or frustrated,the system can generate output actions to relax the user. For instance,the output action can include initiating a driving challenge for theuser or reserving a charging station for the user.

Example Method for Identifying and Performing Output Actions in aVehicle

In some embodiments, the present embodiments relate to performing one ormore output actions relating to a modification of at least one featureof a vehicle. For example, an output action can provide modification tovarious entertainment features (e.g., playback of media), safetyfeatures (e.g., lane assist features), and/or comfort features of avehicle (e.g., interior environment features).

FIG. 12 is a block diagram of an example method 1200 for performing oneor more output actions relating to a modification of at least onefeature of a vehicle. The method may include retrieving a profile modelthat corresponds to a user and includes a series of rules that associatevarious input parameters to corresponding output actions (block 1202).The profile model can be processed to derive various output actionsbased on input information. For example, the profile model can beutilized to modify an internal temperature of the vehicle responsive todetermining exterior temperature changes. As another example, theprofile model can include a rule indicating that at 6 am, the vehicledestination should be routed to the workplace of the user.

The user profile may include a set of user-specific characteristics thatcorrespond to modifications to settings of the vehicle that are specificto the user. The set of user-specific characteristics can modify theseries of rules that associate various input parameters to correspondingoutput actions according to the set of user-specific characteristics.The user profile can include a set of user-specific biometric featuresthat correspond to various emotional states. The various emotionalstates can include a series of predetermined emotional states withspecific features that correspond to each state. For example, detectingthat the user is smiling can correspond to a “happy” emotional state.

In some embodiments, the method may include deriving the set ofuser-specific characteristics in the user profile by comparing the dataacquired by the series of sensors with a series of predetermined profiletypes to identify a predetermined profile type that corresponds to theset of user-specific characteristics. Each predetermined profile typecan include characteristics or tendencies common among a subset ofindividuals. For instance, a predetermined profile type can include a“car enthusiast” profile type that includes features that place anemphasis on utilizing performance features of the vehicle.

In some embodiments, the method may include identifying an identity ofthe user located within the vehicle. The profile model can be retrievedresponsive to identifying the identity of the user. The method can alsoinclude detecting other objects in the vehicle, such as cargo or pets,for example, using the techniques as described herein.

In some embodiments, identifying the identity of the user is based onany of comparing biometric data included in the data acquired by theseries of sensors with a listing of biometric data of a series of knownusers and receiving an identification message from a mobile devicesassociated with the user that is indicative of the identity of the user.

In some embodiments, the method may include identifying a region in thevehicle in which the identified user is located. The one or more outputactions can be performed within the identified region. For instance,upon identifying a user in a passenger seat region, climate controlfeatures in the passenger seat region can be modified corresponding tothe profile model.

The method may include inspecting data acquired by a series of sensorsto identify vehicle environment characteristics indicative of a statusof the vehicle environment (block 1204). The series of sensors caninclude any suitable number of sensors disposed exterior or interior ofthe vehicle as described herein. The data acquired by the sensors can beprocessed to identify vehicle environment characteristics (e.g.,entertainment playback settings, safety feature operation settings,internal climate control settings).

The method may include inspecting the data acquired by the series ofsensors to identify a deviation between a set of baseline biometriccharacteristics and a set of obtained biometric characteristics that isindicative of an emotional state of the user (block 1206). The set ofbaseline biometric characteristics can include a set of universalbiometric characteristics common among a plurality of individuals. Forexample, a baseline biometric characteristic can include facial featuresthat indicate that the user has a neutral emotional state. The baselinebiometric characteristics can be compared to find a derivation with theobtained biometric characteristics of the user. The obtained biometriccharacteristics can include detected features of the user, such asfacial features of the user, a tone of the user's voice, a heart rate ofthe user, language used by the user, etc. The deviation between the setof baseline biometric characteristics and the set of obtained biometriccharacteristics can be indicative of an emotional state of the user.

The method may include processing the identified vehicle environmentcharacteristics and the deviation between the set of baseline biometriccharacteristics and the set of obtained biometric characteristics usingthe profile model to generate one or more output actions capable ofbeing performed on the vehicle (block 1208).

In some embodiments, processing the identified vehicle environmentcharacteristics and the deviation between the set of baseline biometriccharacteristics and the set of obtained biometric characteristics usingthe profile model further comprises comparing the series of rulesincluded in the profile model with a series of vehicle environmentcharacteristics identified from the data acquired by the series ofsensors to generate a first series of recommended output actions, andmodifying the first series of recommended output actions to include onlythe one or more output actions based on features derived from thedeviation between the set of baseline biometric characteristics and theset of obtained biometric characteristics.

The method may include performing the one or more output actions on thevehicle (block 1210). Output actions can modify various components inthe vehicle. For example, various entertainment features, safetyfeatures, and/or comfort features can be modified in the output actions.

In some embodiments, the method may include presenting the one or moreoutput actions capable of being performed on the vehicle to the user andreceiving an indication to perform a number of the one or more outputactions presented to the user, wherein only the number of output actionsindicated by the indication are performed on the vehicle.

In some embodiments, the method may include presenting a request forfeedback to the user relating to the one or more output actionsperformed on the vehicle and receiving a response from the userincluding an indication of whether the one or more output actionsperformed on the vehicle was appropriate. The profile model may beupdated to include the identified vehicle environment characteristicsand the deviation between the set of baseline biometric characteristicsand the one or more output actions performed on the vehicle.

In some embodiments, the method may include determining that a routeddestination of the vehicle is within a threshold distance of an externaldevice environment associated with the user. The method may alsoinclude, responsive to determining that the routed destination of thevehicle is within the threshold distance of the external deviceenvironment associated with the user, sending an indication to theexternal device environment including a request to perform outputactions that correspond to the one or more output actions performed onthe vehicle.

In some embodiments, the method may include determining an operationalstate of the vehicle that includes an operating state and an idle state,wherein a first subset of output actions are capable of being performedwhile in the operating state and a second subset of output actions arecapable of being performed while in the idle state. For example, thedriver may be unable to access video content or game content while thevehicle is in the operating state. In this example, when the vehicletransitions into the idle state (e.g., when the vehicle is parked, thevehicle is charging), other content, such as video content, may becomeavailable for the user.

In some embodiments, the method may include inspecting other usercharacteristic data to generate recommended actions at various points intime. For example, after one hour of continuous driving, the system canprovide a message requesting confirmation to the user to stop thevehicle and take a break. The system may recommend various destinations(e.g., a restaurant, coffee shop, gym) to stop. This may includeutilizing predictive analysis to estimate a user emotional state basedon previously identified emotional state characteristics of the user.For instance, if the user is typically frustrated after work at 6:00 pm,the system can predictively determine that the user is likely to befrustrated and can enable output actions to attempt to relax the user.

In some embodiments, the microphone disposed in the vehicle can be usedto determine a noise level in the vehicle. For example, if themicrophone detects audio with a greater volume, the user may be morelikely to be upset or angry, as the greater volume noises can frustratethe user. The noise level information can be used in detecting theemotional state of the user.

Presentation of User-Specific Entertainment Content in a Vehicle

As noted above, various entertainment features may be available in avehicle. For instance, components in a vehicle can be configured toplayback entertainment content. The playback of entertainment contentcan be user-specific. For example, audio content that is identified bythe user can be played back on speakers in the vehicle.

Further, the present embodiments may relate to user-specific output ofentertainment content by processing a user profile model and variousinput parameters. For example, an emotional state of the user can bedetected. In this example, the emotional state of the user can beprocessed using the user profile model to generate user-specificmodifications to entertainment content in the vehicle.

FIG. 13 is a block diagram illustrating functional modules executed bythe entertainment system 210, according to some embodiments. As a partof the entertainment content that can be provided by the entertainmentsystem 210, the entertainment content can select and output mediacontent to a passenger in the vehicle. As shown in FIG. 13, theentertainment system 210 can execute a media player 1310, aninput/output interface 1320, and a media selection module 1330. Themodules shown in FIG. 13 may comprise software executed by a processorassociated with the entertainment system 210, hardware (such as aspecial-purpose circuit), or a combination of software and hardware.Other embodiments of the entertainment system 210 can includeadditional, fewer, or different modules, or may distribute functionalitydifferently between the modules.

The media player 1310 retrieves media content and presents the mediacontent for display to the driver via the heads-up display 115. Themedia player 1310 may be any type of system capable of presenting mediacontent, such as a video player, a music player, a gaming system, aspecial-purpose application such as a business, productivity, or socialmedia application, or a web browser. The media player 1310 cancommunicate with the remote servers 150 to retrieve media content forpresentation to the driver or passenger in the vehicle 110. In somecases, the media player 1310 may include one or more storage devices, orcan be communicatively coupled to a storage device in the vehicle 110 orremovably coupled to the vehicle 110, that store media content forpresentation to the users. In some embodiments, the entertainment system210 may include multiple media players 1310, each dedicated topresenting a type of media content (where the presentation can include,for example, depending on the type of the media content, generatinginterfaces for rendering by the heads-up display 115, outputting audiostreams by one or more speakers in the vehicle 110, or receiving inputsto control the media content from the input devices 105).

In some embodiments, where the vehicle 110 is an electric vehicle thatperiodically needs to be recharged, the media player 1310 provides mediain response to detecting that the vehicle 110 has been connected to acharger to recharge the vehicle's battery. For example, the media player1310 can receive a signal when the vehicle has begun charging, andinitiate playback of media content in response to the signal. The mediaplayer 1310 can provide the media content throughout the duration of thevehicle charging process.

The input/output (I/O) interface 1320 provides an interface between theentertainment system 210 and input or output devices associated with thevehicle 110. The I/O interface 1320 can receive signals from inputdevices and process the signals to control respective aspects of theentertainment system 210. Similarly, the I/O interface 1320 can generateoutput signals and send the signals to output devices to cause an outputby the output devices. Output devices in the vehicle 110 can include anyof a variety of devices that output signals detectable to the driver,including visual outputs (e.g., the heads-up display 115), audibleoutputs (e.g., a speaker), tactile outputs (e.g., an input device thatprovides haptic feedback, a vibration device in the driver's seat thatcauses the driver's seat to move in association with media content, or amassaging device in the driver's seat that massages the driver), orolfactory outputs (e.g., a device that mists a scent into the air insidethe vehicle 110).

The media selection module 1330 selects media to present to the driver.In some cases, the media selection module 1330 selects media in responseto a specific request from the driver. For example, the driver selectsthe media by providing voice input, gesture input, selecting the desiredmedia from a menu, or the like, and the media selection module 1330retrieves the selected media for the media player 1310 to display to thedriver. In other cases, the media selection module 1330 selects mediafrom a preset option. For example, the driver may set a rule to alwaysplay a certain game when the vehicle 110 is charging. As anotherexample, the media selection module 1330 may apply a rule to resumecontent the driver was consuming on another device, such as resumingplayback of a video the driver was watching on a home television orresuming review of a document the driver was drafting on a workcomputer.

In some embodiments, the media selection module 1330 trains a model toselect media content for the driver. The model can be trained using dataindicating parameters of media content previously provided to the driverand a response of the driver to the provided media content. The modelcan be generated to output a selection of media content to achieve adesired driver response.

In some embodiments, the model can be trained based on whether thedriver changes automatically selected media content to a different typeof content, such that the model can be used to determine if a driver islikely to change the media content if specified content is selected. Forexample, the model may output a determination that the driver alwayschanges a dramatic movie to a comedic movie if the entertainment system210 selects a dramatic movie to play to the driver. Other features canbe used to train the model in addition to an identifier of the selectedmedia content and an indicator of whether the driver changed theselected content, such as a time of day, the driver's schedule, orlocation of the vehicle. For example, the model may output adetermination that the driver usually launches an email application ifit is a weekday but usually does not switch away from video content ifit is a weekend.

In some embodiments, the model can be trained based on an emotionalstate of the driver, using the driver's emotional state either as aninput to select media content (e.g., given that the driver has aspecified emotional state, select certain media content that will appealto the driver in this state), a desired driver response to the mediacontent (e.g., select media content that will cause the driver to reacha specified emotional state), or both. For example, if the driver isdetermined to be fatigued, applying the model may cause the mediaselection module 1320 to select an exercise program as the media contentto present to the driver based on a determination that exercise willenergize the driver for the remaining of her drive (changing thedriver's emotional state from a first state—fatigue—to a secondstate—energized). As another example, if the driver is determined to bestressed, applying the model may cause the media selection module 1320to select calming music and switch the driver's seat to a massage mode.The driver's emotional state can be determined as described above.

In some embodiments, the media selection module 1330 determines amodality for presenting media content to the driver, in addition toselecting the content to present. The modality can be determined basedon factors such as the type of content, whether the vehicle is moving,location of the vehicle, or the emotional state of the driver. Forexample, if the media content is a movie that the driver will watchwhile the vehicle 110 is charging, the media selection module 1330 mayselect to display the movie across an entirety of (or a substantialportion of) the vehicle's windshield to immerse the driver in the movie.If the driver is instead presented media content that either requires orbenefits from contextual awareness, such as an augmented reality gamethat relies upon the player being able to see an area outside thevehicle, the media selection module 1330 may display the media contentwithin a smaller area that does not block the driver's view of theoutside area.

FIG. 14 is a flowchart illustrating an example process 1400 forselecting entertainment content for output to a passenger in a vehicle.In the context of FIG. 14, the passenger can include a person who is atleast partially involved in driving the vehicle (e.g., a driver) as wellas any passive riders in the vehicle. The process 1400 can be performedby the vehicle, for example by the entertainment system 210 in thevehicle. Other embodiments of the process 1400 can include additional,fewer, or different steps, and the steps can be performed in differentorders.

As shown in FIG. 14, the vehicle receives, at step 1402, a state of apassenger in the vehicle. The state of the passenger can include atleast an emotion of the passenger. The state of the passenger canadditionally or alternatively include information about the position ofthe passenger in the vehicle or an indication of whether the passengeris actively driving the vehicle.

At step 1404, the vehicle receives a context of the vehicle. The contextcan include, for example, whether the vehicle is moving and, if it ismoving, how quickly the vehicle is traveling, a mode of operation of thevehicle (e.g., whether it is operated in a self-driving or manuallydriven mode), or the like. The context can additionally or alternativelyinclude an indication of a condition in an environment around thevehicle, such as a time of day, a location of the vehicle, a type ofroad on which the vehicle is traveling (if the vehicle is moving), anamount of traffic on the road, or weather at the vehicle's location.

At step 1406, the vehicle applies an entertainment selection model to atleast the passenger state to select entertainment content for thepassenger. The entertainment selection model can include a series ofinput parameters and corresponding output parameters that can becompared with the passenger state to select a selected output parameterthat represents selected entertainment content for the passenger.

At step 1406, the vehicle enables output of the entertainment content tothe passenger. When the output of the entertainment content is enabled,a passenger can request to receive the entertainment content and theentertainment system 210 will output the entertainment content uponreceiving the passenger's request. In contrast, if a passenger requestsentertainment content that is not enabled, the entertainment system 210may deny the request or output a different type of content.

In some embodiments, the vehicle may enable the entertainment content ifthe context of the vehicle satisfies a condition. For example,entertainment content may be enabled if the vehicle is stopped (e.g., ata charging station) but not enabled if the vehicle is in motion. Asanother example, entertainment content may be enabled if the vehicle iscurrently operated in a self-driving mode, or if the vehicle istraveling on a certain type of road. Alternatively, some types ofentertainment content can be enabled if the vehicle is stopped oroperated in a self-driving mode, while other types of entertainmentcontent are enabled if the vehicle is moving and the passenger isperforming at least one of the driving functions of the vehicle. Forexample, output of music or audiobook recordings may be enabled underany vehicle context, while a video game may only be enabled when thevehicle is parked.

In other embodiments, the vehicle may enable the entertainment contentif the passenger's state satisfies a condition. For example,entertainment content may be always enabled for passengers seated in thebackseat of a car or for passengers on an airplane. As another example,if the vehicle determines that a driver is more focused on the road whenlistening to music than when listening to an audiobook, the vehicle mayenable music while disabling access to audiobooks if the driver isdistracted.

At step 1408, the vehicle outputs the selected entertainment content tothe passenger. In some embodiments, the entertainment content is outputin response to a request by the passenger. For example, when approachinga charging station, the vehicle may ask the passenger whether he wouldlike to resume playing a selected television show while his vehiclecharges. If the passenger confirms, the vehicle can play the selectedtelevision show during the charging process. In other embodiments, theentertainment content is output automatically. For example, if aparticular type of music is selected for the passenger based on adetermination that the passenger is stressed and the music will mitigatethe passenger's stress, the vehicle may automatically begin playing themusic upon the content's selection.

Example Media System Uses

As a first example use of the entertainment system 210, the system canbe used to launch a game while the vehicle 110 is charging. As thevehicle is pulling into a charging station, the entertainment system 210can prompt the driver to select a game to play (e.g., by asking thedriver if she would like to resume the game she played the last time thevehicle was charging). Once the driver selects a game and the vehiclehas been connected to the charger, the entertainment system 210 canlaunch the game and display images associated with the game on theheads-up display 115. In some cases, other components of the vehicle 110can also be used for inputs or outputs related to the game. For example,the steering wheel of the vehicle may be usable as an input device tocontrol gameplay. The driver's seat may move based on the gameplay, suchas transitioning into a rumble seat that moves or vibrates based on thecontent of or actions taken in the game.

Another example of the entertainment system 210 enables the driver towork while the vehicle 110 is charging. For example, the entertainmentsystem 210 may launch an application related to business orproductivity, such as an email application, a word processingapplication, an integrated development environment, a virtual machine ordigital workspace, or the like. The application can be selected, forexample, by determining a last item the driver accessed in the vehicleor on another device. For example, if the driver was editing a documentin a word processing application running on his work computer beforedriving to the charging station, the entertainment system 210 can launcha word processing application to resume review of the document while thevehicle is charging. As another example, if the driver received one ormore emails while she was driving, the entertainment system 210 maylaunch an email application to enable the driver to read and respond tothe emails.

Yet another example of the entertainment system 210 provides a guidedfitness routine that the driver can complete next to the vehicle whilethe vehicle is charging. For example, if the entertainment system 210determines that the driver is fatigued and needs to be energized beforecontinuing to drive, the entertainment system 210 may select a fitnessroutine that instructs the driver to complete one or more exercises.Providing the fitness routine can include displaying a video showing howto perform each exercise in the routine, playing audio describing theexercises, and/or displaying text with instructions for the exercises.

Still another example use of the entertainment system 210 providessocial media functionality to facilitate interactions between driversand passengers at a charging station. For example, a driver can beassociated with a profile on a social network. If the driver opts in tothe social networking functionality when the driver is stopping at acharging station, the driver's profile can be displayed to other driversat the charging station via the entertainment system 210 in theirvehicles. The driver can also view profiles of the other drivers, aswell as optionally the profiles of passengers in the vehicles. Thedrivers at a charging station at a given time can therefore learn abouteach other and interact with one another based on the profiles.Presenting the profiles may include, for example, notifying two driversat a charging station at the same time that they share a commoninterest, based on the profiles associated with the two driversincluding the interest. The drivers can then choose to interact with oneanother based on the common interest. Presenting the profiles mayalternatively include connecting two drivers to compete against eachother in a game while they wait for their vehicles to charge. In stillanother example, the social media functionality can include a datingservice. Drivers or passengers who sign up for the dating service canview dating profiles of other drivers or passengers who have signed forthe service and who are currently charging their vehicles at thecharging station. In some cases, when providing the social mediafunctionality, the entertainment system 210 can facilitate electroniccommunication between users of the social media services, such asconnecting the users to chat electronically while the users remain intheir vehicles. In other cases, the entertainment system 210 canfacilitate face-to-face communication. For example, the entertainmentsystem 210 can provide information that helps a driver or passenger findanother user of the system, such as displaying a picture or descriptionof the user, displaying a picture or description of the vehicle drivenby the user, or causing one or both vehicles to output a specified lightor sound.

In some cases, the entertainment system 210 can select and provide mediacontent while the vehicle 110 is in motion, in addition to the selectionof media content while the vehicle is charging. As an example, theentertainment system 210 can select a type of navigational content todisplay on the heads-up display 115 while the driver is driving thevehicle 110. Navigational content that gamifies navigation, such asachieving a shortest distance to a destination, achieving a shortesttime to the destination, traveling on different roads than usual, orcounting a number of a certain type of vehicle or landmark passed, canbe selected based on factors such as the driver's emotional state orschedule. For example, if the driver is fatigued, gamified navigationalcontent can be selected that will engage the driver in the process ofdriving to ensure that he is awake and alert. As another example, if thedriver is on her way to a meeting but has time to spare before themeeting starts, gamified or informational navigational content may beselected that takes the driver away from the most efficient route to themeeting, but entertains the driver until the start of the meeting. Thenavigational content can, for example, route the driver through ahistorical area near her destination and provide the driver withinformation about the history of the area as she drives through it. Ifthe driver is stressed and has limited time to reach a destination, theentertainment system 210 can instead display normal navigational contentthat will route the driver to the destination without overstimulation.

In other examples of providing media content while the vehicle 110 is inmotion, the entertainment system 210 can automatically select a type ofmusic to play based on the driver's mood, the driver's schedule, thedriver's location, or other factors. The entertainment system 210 canalso control how information is displayed via the heads-up display 115based on the driver's emotional state, time of day, traffic conditions,location, or other factors. For example, the entertainment system 210can display more information on the heads-up display 115 when the driveris more alert and traffic is light, whereas less information may bedisplayed if the driver is unfocused and driving through apedestrian-heavy portion of a city.

In some cases, providers of entertainment content may specify one ormore parameters of the vehicle that can be automatically applied to thevehicle or recommended to a passenger when accessing the entertainmentcontent. For example, if the entertainment content is a video, the videoprovider may specify parameters for the lighting inside the vehicle,volume of the content at various times, brightness of a display devicedisplaying the video, size of the display area, tactile outputs (such asseat vibrations), or olfactory outputs (such as aromatherapy scentsemitted throughout the video or at designated times). The provider ofthe entertainment content can specify the desired parameters through anAPI of the vehicle experience system 310.

Implementation of User-Specific Comfort-Related Output Actions in aVehicle

Many vehicle environments include components that facilitate control offeatures of an internal environment of the vehicle. For example, aninternal climate (e.g., temperature, fan speed) of a vehicle can becontrolled by a user.

Further, each user in a vehicle generally has a unique set of desiredsettings for an internal environment of the vehicle. For example, afirst user may have a desired internal climate temperature of a firsttemperature, while a second user has a different desired internalclimate temperature that includes a much cooler temperature.Additionally, each user in a vehicle may have differing desired internalenvironment settings based on various factors, such as a time of day,outside conditions, an emotional state of the user, etc. While internalenvironment settings are discussed as example comfort-related outputactions, any suitable type of comfort-related output actions can begenerated.

Accordingly, the present embodiments may relate to generation andimplementation of user-specific comfort-related output actions in avehicle. A user-specific profile model can be used to process inputinformation to generate output actions that modify an internalenvironment of the vehicle that correspond to desired settings by theuser. For example, the user-specific profile model may process adetected emotional state of the user and various vehicle environmentalcharacteristics to generate one or more output actions.

FIG. 15 is a block diagram of an example method 1500 for generating andimplementing user-specific comfort-related output actions in a vehicle.The method may include receiving a state of a passenger in a vehicle(block 1502). A state of a passenger can include an emotional state ofthe passenger, for example.

The method may include receiving a context of the vehicle (block 1504).The context of the vehicle can include modifications to various settingsof the vehicle. For example, the context of the vehicle can include amodification to a fan intensity setting.

The method may include applying the passenger state and vehicle contextinformation to a comfort personalization model configured to select atleast one comfort parameter for the passenger (block 1506). The at leastone comfort parameter can include an output action that modifies acomfort-related component in the vehicle, such as a fan or airconditioner, for example.

The method may include causing the vehicle to output the selectedcomfort parameter (block 1508). This may include modifying a setting ofthe vehicle relating to the comfort of the user. For example, causingthe vehicle to output the selected comfort parameter can includemodifying the internal temperature of the vehicle.

Example Method for Performing One or More Output Actions Relating to aModification to at Least One Internal Environmental Feature of a Vehicle

In some embodiments, a method for performing one or more output actionsrelating to a modification to at least one internal environmentalfeature of a vehicle includes retrieving a user profile model thatcorresponds to a user.

In some embodiments, the method may include identifying an identity ofthe user located within the vehicle, wherein the profile model isretrieved responsive to identifying the identity of the user, whereinidentifying the identity of the user is based on any of comparingbiometric data included in the data acquired by the series of sensorswith a listing of biometric data of a series of known users andreceiving an identification message from a mobile device associated withthe user that is indicative of the identity of the user.

The method may include inspecting data acquired by a series of sensorsdisposed on the vehicle to identify a series of vehicle environmentcharacteristics indicative of characteristics of an internal environmentof the vehicle;

The method may include inspecting the series of sensors disposed on thevehicle to identify an emotional state of the user. In some embodiments,identifying the emotional state corresponding to the user furthercomprises inspecting the data acquired by the series of sensors toidentify a deviation between a set of baseline biometric characteristicsand a set of obtained biometric characteristics that is indicative ofthe emotional state of the user, and comparing the deviation between theset of baseline biometric characteristics and the set of obtainedbiometric characteristics with a series of predetermined emotionalstates to identify a predetermined emotional state with features thatcorrespond to the deviation between the set of baseline biometriccharacteristics and the set of obtained biometric characteristics.

The method may include processing the identified series of vehicleenvironment characteristics and the identified emotional state using theuser profile model to generate the one or more output actions includingmodifications to at least one internal environmental feature of thevehicle.

In some embodiments, processing the identified series of vehicleenvironment characteristics and the identified emotional state using theuser profile model further comprises identifying a subset of inputactions that are within a threshold similarity of the vehicleenvironment characteristics, and identifying a number of the subset ofinput actions that correspond to the data acquired by the series ofsensors disposed on the vehicle, wherein the one or more output actionsinclude output actions that correspond to each of the number of thesubset of input actions.

The method may include performing the one or more output actions on thevehicle. In some embodiments, the one or more output actions includemodifying an internal temperature of the vehicle, modifying a fan speed,modifying an operational state of an aromatherapy diffuser, andmodification an operational state of a seat massaging device.

In some embodiments, the method may include presenting a request forfeedback to the user relating to the one or more output actionsperformed on the vehicle, and receiving a response from the userincluding an indication of whether the one or more output actionsperformed on the vehicle was appropriate, wherein the user profile modelis updated to include the vehicle environment characteristics, theemotional state of the user, and the one or more output actions.

In some embodiments, the method may include determining that a routeddestination of the vehicle is within a threshold distance of an externaldevice environment associated with the user, and responsive todetermining that the routed destination of the vehicle is within thethreshold distance of the external device environment associated withthe user, sending an indication to the external device environmentincluding a request to perform output actions that correspond to the oneor more output actions performed on the vehicle.

In some embodiments, processing the identified vehicle environmentcharacteristics and the deviation between the set of baseline biometriccharacteristics and the set of obtained biometric characteristics usingthe profile model comprises identifying a subset of input actions thatare within a threshold similarity of the vehicle environmentcharacteristics, and identifying a number of the subset of input actionsthat correspond to the deviation between the set of baseline biometriccharacteristics and the set of obtained biometric characteristics,wherein the one or more output actions include output actions thatcorrespond to each of the number of the subset of input actions.

Implementation of User-Specific Safety-Related Output Actions in aVehicle

Many vehicle environments include components that facilitate control ofsafety features of the vehicle. For example, safety features can includea lane-assist feature, a distance control feature, modifying a volume ofaudio in the vehicle, etc.

The modification of the safety features can be user-specific and/orbased on the emotional state of the user. For example, upon determiningthat a user is frustrated or angry, a lane assist feature may beinitiated to assist the user in maintaining a lane position of thevehicle.

The present embodiments may relate to generation and implementation ofuser-specific safety-related output actions in a vehicle. Auser-specific profile model can be used to process input information togenerate output actions that modify safety features of the vehicle.

FIG. 16 is a block diagram of an example method 1600 for generating andimplementing user-specific safety-related output actions in a vehicle.The method may include receiving a state of a passenger in a vehicle(block 1602). This may include an emotional state of the user.

The method may include receiving a context of the vehicle (block 1604).The context may include characteristics of the environmentinterior/exterior to the vehicle and/or user characteristics.

The method may include applying the passenger state and the vehiclecontext to a safety personalization model configured to select at leastone safety parameter for the passenger (block 1606). The at least onesafety parameter may include modifying safety features of the vehicle.The method may include causing the vehicle to enact a safety responsebased on the selected safety parameter (block 1608).

The safety response can be user-specific, as the personalized safetyresponse plan can be based on the emotion detection processes,personalized models, or other systems, processes, or data structuresdescribed herein.

Example Method for Performing One or More Output Actions Relating to aModification to at Least One Vehicle Safety Feature of a Vehicle

In some embodiments, a method for performing one or more output actionsrelating to a modification to at least one safety feature of a vehicleincludes retrieving a user profile model that corresponds to a user. Theuser profile model can include a series of input actions and a series ofcorresponding output actions that are unique to the user.

In some embodiments, the method can include retrieving usercharacteristic information that is indicative of user-specificmodifications to safety features of the vehicle. The method may alsoinclude comparing the retrieved user characteristic information with aseries of predetermined profile types to identify a predeterminedprofile type with features that correspond to the retrieved usercharacteristic information. The method may also include updating theuser profile model with the features associated with the predeterminedprofile type that corresponds to the retrieved user characteristicinformation.

The method can include inspecting a set of data acquired by a series ofsensors disposed on a vehicle to identify a series of vehicleenvironment characteristics indicative of a status of safety features ofthe vehicle and a set of user environmental characteristics relating tofeatures of the user.

The method can include retrieving a series of predetermined emotionalstates, each predetermined emotional state including a plurality offeatures indicative of an emotional state.

The method can include comparing the user environmental characteristicswith the series of predetermined emotional states to identify apredetermined emotional state with features that correspond to the userenvironmental characteristics. An emotional state of the user mayinclude the predetermined emotional state.

The method can include processing the identified series of vehicleenvironment characteristics and the identified emotional state using theuser profile model to generate the one or more output actions that areindicative of one or more modifications to any of a series of safetyfeatures of the vehicle.

The method can include causing execution of the one or more outputactions on the vehicle. In some embodiments, the method can includepresenting a request for feedback to the user relating to the one ormore output actions performed on the vehicle. In some embodiments, theoutput actions include any of initiation of a lane position maintenanceprocess, increasing a minimum distance between the vehicle and anadjacent vehicle by a vehicle distance maintenance process, and areduction of volume of any entertainment content outputting in thevehicle.

The method may also include receiving a response from the user includingan indication of whether the one or more output actions performed on thevehicle was appropriate. The user profile model may be updated toinclude the vehicle environment characteristics, the emotional state ofthe user, and the one or more output actions.

In some embodiments, the method can include determining an operationalstate of the vehicle that includes an operating state and an idle state.Only a subset of output actions may be allowed to be performed when inthe operating state. The method may also include determining that theoperational state of the vehicle has transitioned from the idle state tothe operating state. The method may also include, responsive todetermining that the operational state of the vehicle has transitionedfrom the idle state to the operating state, presenting a series ofrecommended output actions to the user that are capable of beingperformed in the transitioned operational state. The method may includeperforming a number of the recommended output actions responsive toobtain a response from the user indicating a confirmation to perform thenumber of recommended output actions.

In some embodiments, the method can include determining that a routeddestination of the vehicle is within a threshold distance of an externaldevice environment associated with the user, and responsive todetermining that the routed destination of the vehicle is within thethreshold distance of the external device environment associated withthe user, send an indication to the external device environmentincluding a request to perform output actions that correspond to the oneor more output actions performed on the vehicle.

Example Processing System

FIG. 17 is a block diagram illustrating an example of a processingsystem 1700 in which at least some operations described herein can beimplemented. For example, the remote server 120 may be implemented asthe example processing system 1700, and various systems associated withthe vehicle 110 may include some or all of the components shown in FIG.17. The processing system 1700 may include one or more centralprocessing units (“processors”) 1702, main memory 1706, non-volatilememory 1710, network adapter 1712 (e.g., network interfaces), videodisplay 1718, input/output devices 1720, control device 1722 (e.g.,keyboard and pointing devices), drive unit 1724 including a storagemedium 1726, and signal generation device 1730 that are communicativelyconnected to a bus 1716. The bus 1716 is illustrated as an abstractionthat represents any one or more separate physical buses, point to pointconnections, or both connected by appropriate bridges, adapters, orcontrollers. The bus 1716, therefore, can include, for example, a systembus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, aHyperTransport or industry standard architecture (ISA) bus, a smallcomputer system interface (SCSI) bus, a universal serial bus (USB), IIC(I2C) bus, or an Institute of Electrical and Electronics Engineers(IEEE) standard 1794 bus, also called “Firewire.”

In various embodiments, the processing system 1700 operates as part of auser device, although the processing system 1700 may also be connected(e.g., wired or wirelessly) to the user device. In a networkeddeployment, the processing system 1700 may operate in the capacity of aserver or a client machine in a client-server network environment, or asa peer machine in a peer-to-peer (or distributed) network environment.

The processing system 1700 may be a server computer, a client computer,a personal computer, a tablet, a laptop computer, a personal digitalassistant (PDA), a cellular phone, a processor, a web appliance, anetwork router, switch or bridge, a console, a hand-held console, agaming device, a music player, network-connected (“smart”) televisions,television-connected devices, or any portable device or machine capableof executing a set of instructions (sequential or otherwise) thatspecify actions to be taken by the processing system 1700.

While the main memory 1706, non-volatile memory 1710, and storage medium1726 (also called a “machine-readable medium) are shown to be a singlemedium, the term “machine-readable medium” and “storage medium” shouldbe taken to include a single medium or multiple media (e.g., acentralized or distributed database, and/or associated caches andservers) that store one or more sets of instructions 1728. The term“machine-readable medium” and “storage medium” shall also be taken toinclude any medium that is capable of storing, encoding, or carrying aset of instructions for execution by the computing system and that causethe computing system to perform any one or more of the methodologies ofthe presently disclosed embodiments.

In general, the routines executed to implement the embodiments of thedisclosure, may be implemented as part of an operating system or aspecific application, component, program, object, module or sequence ofinstructions referred to as “computer programs.” The computer programstypically comprise one or more instructions (e.g., instructions 1704,1708, 1728) set at various times in various memory and storage devicesin a computer, and that, when read and executed by one or moreprocessing units or processors 1702, cause the processing system 1700 toperform operations to execute elements involving the various aspects ofthe disclosure.

Moreover, while embodiments have been described in the context of fullyfunctioning computers and computer systems, those skilled in the artwill appreciate that the various embodiments are capable of beingdistributed as a program product in a variety of forms, and that thedisclosure applies equally regardless of the particular type of machineor computer-readable media used to actually effect the distribution. Forexample, the technology described herein could be implemented usingvirtual machines or cloud computing services.

Further examples of machine-readable storage media, machine-readablemedia, or computer-readable (storage) media include, but are not limitedto, recordable type media such as volatile and non-volatile memorydevices 1710, floppy and other removable disks, hard disk drives,optical disks (e.g., Compact Disk Read-Only Memory (CD ROMS), DigitalVersatile Disks (DVDs)), and transmission type media, such as digitaland analog communication links.

The network adapter 1712 enables the processing system 1700 to mediatedata in a network 1714 with an entity that is external to the processingsystem 1700 through any known and/or convenient communications protocolsupported by the processing system 1700 and the external entity. Thenetwork adapter 1712 can include one or more of a network adaptor card,a wireless network interface card, a router, an access point, a wirelessrouter, a switch, a multilayer switch, a protocol converter, a gateway,a bridge, bridge router, a hub, a digital media receiver, and/or arepeater.

The network adapter 1712 can include a firewall which can, in someembodiments, govern and/or manage permission to access/proxy data in acomputer network, and track varying levels of trust between differentmachines and/or applications. The firewall can be any number of moduleshaving any combination of hardware and/or software components able toenforce a predetermined set of access rights between a particular set ofmachines and applications, machines and machines, and/or applicationsand applications, for example, to regulate the flow of traffic andresource sharing between these varying entities. The firewall mayadditionally manage and/or have access to an access control list whichdetails permissions including for example, the access and operationrights of an object by an individual, a machine, and/or an application,and the circumstances under which the permission rights stand.

As indicated above, the techniques introduced here implemented by, forexample, programmable circuitry (e.g., one or more microprocessors),programmed with software and/or firmware, entirely in special-purposehardwired (i.e., non-programmable) circuitry, or in a combination orsuch forms. Special-purpose circuitry can be in the form of, forexample, one or more application-specific integrated circuits (ASICs),programmable logic devices (PLDs), field-programmable gate arrays(FPGAs), etc.

From the foregoing, it will be appreciated that specific embodiments ofthe invention have been described herein for purposes of illustration,but that various modifications may be made without deviating from thescope of the invention. Accordingly, the invention is not limited exceptas by the appended claims.

What is claimed is:
 1. A system comprising: a sensor interfacecommunicatively coupled to a plurality of sensors in a vehicle, thesensor interface comprising: an input receiving data from each of theplurality of sensors; and an output configured to output fused vehicledata based on the data received from the plurality of sensors; and avehicle experience system coupled to the output of the sensor interfaceto receive the fused vehicle data, the vehicle experience systemcomprising one or more processors and a non-transitory computer readablestorage medium storing instructions that when executed by the one ormore processors cause the one or more processors to control at least oneparameter of the vehicle based on the fused vehicle data.
 2. The systemof claim 1, wherein the input is coupled to a car network that iselectronically coupled to the plurality of sensors.
 3. The system ofclaim 2, wherein the car network comprises a controller area network(CAN) bus, a local interconnect network (LIN), or a comfort-CAN bus. 4.The system of claim 2, wherein the vehicle experience system is coupledto the car network, and wherein controlling the at least one parameterof the vehicle comprises outputting the parameter to a vehicle controldevice over the car network.
 5. The system of claim 1, wherein the inputreceives a continual stream of data from each of the plurality ofsensors.
 6. The system of claim 1, further comprising a networkinterface communicatively coupled to one or more remote computingdevices, and wherein the sensor interface is further configured to fusethe vehicle data with data received from the one or more remotecomputing devices via the network interface.
 7. The system of claim 6,wherein the sensor interface is further configured to receive processeddata from the vehicle experience system and output the processed data tothe one or more remote computing devices over the network interface. 8.The system of claim 7, wherein the instructions executed by the one ormore processors cause the one or more processors to apply a privacy ruleto the processed data, wherein the sensor interface selectively outputsthe processed data responsive to application of the privacy rule.
 9. Thesystem of claim 6, wherein the data received from the one or more remotecomputing devices comprises an instruction to apply a parameter to thevehicle, wherein the sensor interface outputs the vehicle data fusedwith the instruction, and wherein controlling the at least one parameterof the vehicle based on the fused vehicle data comprises applying theparameter specified in the instructions to the vehicle.
 10. A methodcomprising: receiving data from a plurality of sensors in a vehicle overa sensor interface communicatively coupled to the plurality of sensors;generating a data structure comprising fused vehicle data based on thedata received from the plurality of sensors; and outputting the datastructure to a vehicle control system configured to control at least oneparameter of the vehicle based on the fused vehicle data.
 11. The methodof claim 10, wherein the vehicle control system is configured to applyone or more rules to the fused vehicle data that, when applied, causethe vehicle control system to select the at least one parameter of thevehicle.
 12. The method of claim 10, wherein the vehicle control systemis configured to apply a trained model to the fused vehicle data that,when applied, causes the vehicle control system to select the at leastone parameter of the vehicle.
 13. The method of claim 10, wherein thesensor interface is coupled to a car network that is electronicallycoupled to the plurality of sensors, and wherein the vehicle controlsystem is coupled to the car network to control the at least oneparameter of the vehicle by outputting the at least one parameter to thecar network.
 14. A non-transitory computer readable storage mediumstoring executable computer program instructions comprising anapplication programming interface, the computer program instructionswhen executed by a processor on a vehicle that is communicativelycoupled to a vehicle experience system of the vehicle causing theprocessor to: receive an instruction to apply a parameter to the vehiclefrom a computing device remote from the vehicle; and cause the vehicleexperience system to apply the parameter to the vehicle.
 15. Thenon-transitory computer readable storage medium of claim 14, wherein theinstructions when executed further cause the processor to: receive datafrom each of a plurality of sensors in the vehicle; and generate fusedvehicle data based on the data received from the plurality of sensors.16. The non-transitory computer readable storage medium of claim 15,wherein the instructions when executed further cause the processor to:process the fused vehicle data to generate processed data; and outputthe processed data to the computing device remote from the vehicle. 17.The non-transitory computer readable storage medium of claim 16, whereinthe instructions when executed further cause the processor to: apply aprivacy rule to the processed data; wherein the processed data isselectively output to the computing device remote from the vehicleresponsive to application of the privacy rule.
 18. The non-transitorycomputer readable storage medium of claim 16, wherein the storage mediumstores a privacy profile associated with a user of the vehicle, andwherein the instructions that cause the processor to apply the privacyrule comprises instructions that cause the processor to access theprivacy profile to retrieve the privacy rule.
 19. The non-transitorycomputer readable storage medium of claim 14, wherein the computingdevice remote from the vehicle comprises a media server configured toserve media content to the vehicle for output by the vehicle, andwherein receiving the instruction to apply the parameter to the vehiclecomprises receiving an instruction to apply a parameter to the vehiclein association with the output of the media content.
 20. Thenon-transitory computer readable storage medium of claim 19, wherein theparameter applied to the vehicle in association with the output of themedia content includes a lighting parameter, a volume parameter, adisplay brightness parameter, a display size parameter, a tactile outputparameter, or an olfactory output parameter.